

ICAT 


An Experimental Study Of 
The Effect Of Shared Information On 
Pilot/Controller Re-Route Negotiation 

Todd C. Farley and R. John Hansman 


Report No. ICAT-99-1 
January 1999 


MIT International Center for Air Transportation 

Department of Aeronautics & Astronautics 
Massachusetts Institute of Technology 
Cambridge, MA 02139 USA 



AN EXPERIMENTAL STUDY OF THE EFFECT OF 
SHARED INFORMATION ON PILOT/CONTROLLER 
RE-ROUTE NEGOTIATION 

Todd C. Farley and R. John Hansman 

MIT International Center for Air Transportation 
Cambridge, MA 02139 USA 

January 1999 

ICAT-99-1 





An Experimental Study of the Effect of Shared Information 
on Pilot/Controller Re-Route Negotiation 


Abstract 

Air-ground data link systems are being developed to enable pilots and air traffic controllers to 
share information more fully. The sharing of information is generally expected to enhance then- 
shared situation awareness and foster more collaborative decision making. 

An exploratory, part-task simulator experiment is described which evaluates the extent to which 
shared information may lead pilots and controllers to cooperate or compete when negotiating 
route amendments. The results indicate an improvement in situation awareness for pilots and 
controllers and a willingness to work cooperatively. 

Independent of data link considerations, the experiment also demonstrates the value of providing 
controllers with a good-quality weather representation on their plan view displays. Observed 
improvements in situation awareness and separation assurance are discussed. It is argued that 
deployment of this relatively simple, low-risk addition to the plan view displays be accelerated. 
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Chapter 1 Introduction 


The technology to deliver digital data link communication between aircraft and the 
ground is well developed. Current data link applications include pre-departure clearance delivery 
via Tower Data Link Services (TDLS), global voice and data communications via satellite 
(SATCOM), and weather uplinks via Terminal Weather Information for Pilots (TWIP). The 
FAA's proposed future National Airspace System Architecture (FAA, 1998c) calls for expansion 
of existing data link services to include applications such as the Controller-Pilot Data l ink 
Communication (CPDLC) system. Automatic Dependent Surveillance broadcasts (ADS-B), and 
Aviation Weather Information (AWIN) systems. 

Such advances will allow information which is not uniformly accessible today to be 
shared between pilots, controllers and other users (e.g., dispatchers, airport managers, etc.). This 
sharing of information — an information "party line" — is expected to offer several benefits: 

• Improved shared situation awareness between agents; 

• The ability to better anticipate the needs and/or preferences of other agents; 

• A common informational context upon which to negotiate. 

These benefits ultimately are expected to result in more cooperative interaction between agents, 
moving airspace operations closer to the envisioned goal of Collaborative Decision Making 
(CDM) (Falcone, etal., 1998). 

However, the sharing of information may effect a less desirable outcome, one 
characterized by increased voice communications, increased workload, and increased contention 
between agents. Midkiff & Hansman (1992) found that pilots were more willing to comply with 
air traffic control (ATC) when they knew their own information to be inferior to that of ATC. 
Conversely, they found that pilots were more assertive and willing to question ATC when they 
knew their own information to be equal or superior to that of ATC. 

The implementation of digital data link stands to alter the current "balance of 
information" between ATC and the flight deck. Today, flight crews typically have better weather 
information than air traffic controllers, while air traffic controllers typically have better traffic 
information than the flight crews. These imbalances lend stability to a control system which is 
inherently ambiguous with regard to authority: controllers are responsible for ensuring aircraft 


1 



separation (FAA, 1998b), but pilots are responsible for the operation of their aircraft (FAA, 
1998a). In practice, controllers typically defer to flight crews in matters involving hazardous 
weather. Conversely, flight crews typically defer to air traffic control in matters involving traffic 
conflicts. In effect, authority is assigned implicitly based on the information allocation — the 
agent with the better information assumes authority. Since the information superiority with 
regard to weather and traffic is unambiguous, so too is the decision-making authority in weather- 
and traffic-related situations. 

By sharing weather and traffic information between both parties, data link applications 
will work to redress the current information imbalances. For example, a shared representation of 
convective weather activity may enable controllers to better recognize developing weather 
constraints, anticipate needed deviations, and reorganize the traffic flow earlier and more 
effectively. Similarly, a shared representation of traffic information will provide pilots a fuller 
picture of their surrounding traffic flow, consistent with the information available to ATC. This 
may enable pilots to better anticipate sequencing instructions, correlate PIREPs, and identify 
available route alternatives. 

However, by sharing information, the stability of the system may actually be undermined. 
Sharing information will serve to neutralize any current information advantage — the basis on 
which authority is presently assumed — effectively putting decision-making authority "up for 
grabs". The results of Midkiff & Hansman (1992) suggest that in some situations the availability 
of common information via data link may result in increased negotiation, and with it 
commensurate increases in frequency congestion and workload. In short, it suggests the potential 
for less collaborative, less efficient operations. 

The present study adopted an integrated human-centered systems approach to evaluate 
the effect of an air-ground data link system on pilot-controller interaction. The approach 
considered the human elements of the system as functional components of a closed-loop control 
system (Hansman, et al., 1997). The following four-step process formed the basis of the overall 
system performance assessment: 

Step 1. Determine the goal structures and situation awareness information requirements of 
pilots and air traffic controllers. 
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Step 2. Compare pilots’ and controllers’ goal hierarchies and information requirements to 
identify areas of common, disparate or competing interest. 

Step 3. Based on the findings, design and perform simulator-based test scenarios which 
explore pilots’ and controllers’ interactions and behaviors in environments where 
common information may serve competing goals. 

Step 4. Based on the experimental observations, assess the potential benefits and effects of 
shared information at the system level. 

In order to understand the effect of shared information in the system and how pilots and 
controllers may act on that information, it was necessary first to identify their roles, their motives 
and their informational needs (Step 1). A comprehensive goal-directed task analysis was 
performed for commercial airline pilots (Endsley, Farley, Jones, Midkiff, & Hansman, 1998) to 
complement an existing analysis for en route ATC specialists (Endsley & Rodgers, 1994). The 
two analyses then were compared against one another in order to identify areas of common or 
competing interest between pilots and controllers (Step 2). This effort is detailed in Chapter 2. 

Based on the results of the comparative analysis, an exploratory, part-task simulator 
experiment was conducted to evaluate the extent to which shared information (via air— ground 
data link) may lead pilots and controllers to cooperate or compete when negotiating route 
amendments (Step 3). The experiment paired an airline pilot subject with an ATC specialist 
subject in a real-time simulated air traffic environment. Test scenarios intentionally conflicted 
the goals of pilot and controller in tactical re-routing situations as identified in the comparison of 
the pilot and controller task analyses. Of particular interest were indications of each subject's 
recognition of the other's constraints, anticipation of others’ needs and preferences, willingness 
to comply/cooperate, and persistence in pursuing one’s own preferred solution. The experiment 
is detailed in Chapter 3. 

Chapter 4 presents the experimental results in terms of situation awareness, separation 
assurance, pilot-controller interaction, and workload. 

Chapter 5 summarizes the overall findings of this study (Step 4). It assesses the potential 
for shared information to effect more collaborative or competitive interaction between pilots and 
controllers and discusses the safety implications and low-risk steps to improve the system. 
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Chapter 2 Situation Awareness Analysis 

In order to assess the effects of shared information on pilot-controller interaction, it was 
necessary first to understand their individual interests and information requirements, as well as 
their common ones. Endsley & Rodgers' (1994) analysis of en route air traffic control specialists 
created a goal hierarchy and a list of situation awareness information requirements for the task of 
en route air traffic control. For the present study, a similar methodology was applied to the job of 
commercial airline pilots, facilitating a comparison between the two. 

This chapter begins with a brief introduction to the concept of situation awareness, 
describing the Endsley hierarchical model of situation awareness (SA) and the role of SA in the 
decision-making process. The development of a pilot task analysis is described which parallels 
the existing controller task analysis, identifying the pilot's goal structure and SA information 
requirements. Finally, a comparison of the two analyses is presented along with the conclusions 
it supports. 

2.1 Situation Awareness Model 

Situation awareness is a fundamental requirement for most complex tasks, forming the 
basis for decision making and performance. While there are several acceptable definitions of 
situation awareness, the definition most applicable for this study comes from Endsley (1995): 

"Situation awareness is the perception of the elements in the environment 
within a volume of time and space, the comprehension of their meaning, 
and the projection of their status in the near future." 

The definition casts situation awareness in a hierarchical form At the lowest level, Level 1, is 
the "perception of the elements". This includes the status, attributes, and dynamics of the relevant 
elements of one's environment. Having perceived such elements, the next level of SA is the 
"comprehension of their meaning". This second level of SA requires the individual to integrate 
disjointed Level 1 elements to understand their significance relative to his goals. Inherent in 
Level 2 SA is the ability to recognize the impact of a change in one element on another, the 
ability to detect errors in the system, and the ability to derive rates from raw data. The highest 
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level. Level 3, is the projection of their status in the future". It is the ability to accurately predict 
the future actions of the elements and the future state of the system over the near term. It is this 
level of SA that provides the decision maker with the time to develop and consider alternative 
courses of action. 



Figure 1. Model of situation awareness in decision making (from Endsley, 1995) 

Figure 1 illustrates the role of situation awareness in the decision-making process. While 
decisions may be affected by static factors such as rules, equipment, or training, it is one's 
situation awareness that forms the critical dynamic input to the decision process and subsequent 
action. It is important to recognize that SA is heavily affected by individual factors, principally 
the goals, objectives and preconceptions of the decision maker. 

In air traffic operations, decisions typically involve two or more individuals: pilot, 
controller, dispatcher, etc. The Endsley SA model can be extended to accommodate such "group 
decisions" by incorporating parallel feedback paths, one for each decision maker. Figure 2 
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depicts the model for a pilot-controller decision-making group. Note that each decision maker 
has an independent feedback loop, which may contain different information elements about their 
common environment. Each person interprets his available information (biased by his individual 
goals), acquires some level of SA, and arrives at an initial decision that will support his 
objectives. This initial decision may differ from his counterpart. After a period of negotiation, a 
final decision is reached and an action is taken. The amount of negotiation that takes place is 
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dependent on factors such as the alignment of the negotiators' goals, the relative uncertainty of 
their information, and their respective workload levels. The negotiation period may be brief or 
nonexistent if, for example, both parties arrive at the same initial decision or if one individual's 
high workload precludes negotiation. The negotiation period may be protracted if their goals are 
mutually exclusive, high in priority, and each individual has a high degree of confidence in his 
information. 

The Group SA model illustrates how disparate goals and objectives and/or unequal 
information about the state of the environment may affect the negotiation process by virtue of 
their unique effect on each person's individual level of situation awareness. 

2.2 Comparative SA Analysis Between Pilots and Controllers 

Lewicki, Saunders, & Minton (1997) assert that negotiation arises when two parties have 
competing goals and at least one party believes he can exert some form of influence to gain a 
more favorable resolution. In examining negotiation between pilots and controllers, then, it is 
important to understand where their goals are in common and where they are in competition. 
Furthermore, since information is a common source of power in negotiation (Lewicki, Saunders, 
& Minton, 1997), it is important to consider disparities in their available information and how 
the balance of information may affect the balance of power in pilot-controller negotiations. 

2.2.1 Methodology 


To identify the common and competing goals of controllers and pilots and to determine 
where their respective information advantages he, an analysis was performed comparing the 
goals and information requirements of en route air traffic controllers against those of commercial 
airline pilots. Controller goals and information requirements were taken from a task analysis by 
Endsley & Rodgers (1994). Because no such task analysis existed for commercial airline pilots, 
one was performed as part of this study (Endsley, Farley, Jones, Midkiff, & Hansman, 1998). 

Pilot goals and information requirements were identified by conducting a goal-directed 
task analysis based on elicitation from experienced commercial airline pilots. The analysis was 
performed based on the methodology of Endsley & Rodgers (1994). To start, each expert was 
asked to identify his top-level goal when in command of a commercial airline flight. Once this 
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initial goal was articulated, the remainder of the analysis proceeded in recursive fashion as 
follows. The pilot was asked: 

Step 1. What assessments must be made to accomplish this goal? 

Step 2. What information is required to make those assessments? 

Step 3. What lower-level goals ("sub-goals") contribute to the goal currently under 
consideration? 

These three questions were repeated for each listed sub-goal until that branch of the hierarchy 
was exhausted. The analysis then progressed laterally to develop the remaining branches of the 
hierarchy. 

For example, when assessing the flight plan, pilots specified "avoiding hazardous 
weather as a goal. For step 1 — "What assessments must be made to accomplish that goal?" — 
pilots itemized three basic considerations: 

(a) Is hazardous weather expected en route? 

(b) What is the degree of hazard of anticipated weather conditions? 

(c) Is a change of flight path needed? 

For step 2, pilots identified numerous information items required to make these assessments. For 
instance, with respect to question (a), the information requirements in Table 1 (next page) were 
cited 1 : 


1 The list is hierarchical. Indented items indicate more specific or lower-level information inherent to the higher- 
level information listed above it. 
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Table 1. Example information requirements 


Sub-goal: Avoid hazardous weather 

■ Is hazardous weather expected en route? 

■ Likelihood of hazardous weather encounter 

■ Planned flight path 

■ Current weather pattern and conditions 

■ Area affected 

■ Altitudes affected 

■ Projected weather conditions 

■ Direction and speed of movement 

■ Increasing or decreasing intensity 

■ Confidence level in weather information 


For this example, no subgoals were identified in accordance with step 3, so the analysis moved 
on to repeat step 2 for question (b), and so forth. 

The process to completely specify the entire goal hierarchy with all of its attendant 
assessments and information requirements comprised numerous interview sessions with subject 
matter experts. Each expert was interviewed individually. Interview sessions lasted from one to 
four hours. Two active airline pilots served as subject matter experts. Their average flight 
experience was 9,350 flight hours, encompassing regional, domestic, international and military 
operations. The initial draft was reviewed for accuracy and completeness by six other airline 
pilots (mean flight hours: 10,580). Their comments were incorporated into the final analysis. 

The pilot goal hierarchy is provided in Appendix A; the list of SA information 
requirements is found in Appendix B. The requirements are organized by SA level: perception of 
elements (Level 1), comprehension of their meaning (Level 2), and projection of the future 
(Level 3). 

The completed commercial airline pilot task analysis was then compared against the 
Endsley & Rodgers (1994) air traffic controller task analysis to identify areas in which the goals 
of pilots and controllers could be in competition and to identify the important information 
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elements in those areas. The results were expected to reveal situations in which the sharing of 
information between pilot and controller (via data link) could have an observable effect on 
system stability and efficiency. 

The top levels of the goal hierarchies were compared to identify common elements. The 
structure of each hierarchy was reworked around those common elements to better facilitate 
direct comparisons between the two. The syntax of some elements was revised to be comparable. 
Based on their aligned goals, the underlying assessments and information requirements were 
compared to identify common and unique elements. Common information requirements were 
further examined for their potential to support competing goals between pilots and controllers. 

2.2.2 Results 

Figures 3 and 4 depict the high-level goals of pilots and controllers, respectively. At these 
higher levels, the goal structures are highly parallel, and there is considerable overlap between 
the two. Common interests include such goals as: 

■ Assure flight safety 

■ Avoid conflicts (e.g., aircraft, terrain, restricted airspace) 

■ Assess current and alternate routes 

■ Provide customer service 

■ Handle perturbations (e.g., weather, emergencies) 

■ Manage resources (e.g., people, systems) 

The comparative analysis illustrates the far-reaching effects of re-route decisions. All of 
the first- and second-level goals for both pilots and controllers are influenced by the current and 
future flight path. This suggests that re-route negotiations have broad and significant 

ramifications for both pilots and controllers and that each should have a vested interest in the 
outcome. 

Comparison of the lower-level goals reveals that pilots and controllers often have 
competing interests with respect to re-route decisions. In considering a route deviation, pilots and 
controllers often perform an informal cost/benefit analysis, weighing the benefit in terms of their 
mission objectives against the cost in terms of workload. However, comparison of the task 
analyses indicates that pilots and controllers calculate the costs and benefits differently. 
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Assure safety of flight 


Avoid 

conflicts 


-Maintain separation 
-Avoid restricted 
airspace 

-Avoid obstacles 
and terrain 
-Avoid hazardous 
weather 


Select best 
flight path 


Manage current 
flight plan 
Develop alternate 
routes 

Determine best 
option 


Provide 

customer service 


-Provide ride comfort 
-Adhere to schedule 
-Provide for the safety 
and security of 
passengers 


Handle 

perturbations 


- Minimize impact of 
hazardous weather 

- Respond to 
emergencies 

- Minimize impact of 
abnormal situations 

-Minimize impact of 
abnormal ATC 
situations 


Manage 

resources 


-Tailor operations according 
to colleagues' competency 
and reliability 
-Keep everyone informed 
at appropriate level 
-Tailor operations according 
to the status and reliability 
of the aircraft systems and 
airspace infrastructure 
-Configure aircraft and 
determine system settings 
for phase of flight 
-Configure aircraft and 
determine system settings 
for current weather, terrain 


Figure 3. Commercial airline pilot top-level goal hierarchy 


Assure flight safety 


Maintain separation 
Avoid airspace conflict 
Maintain aircraft 
conformance 
Assure minimum 
altitude requirements 


Manage 
traffic flows 


Manage arrival flows 
Manage departure flows 
Develop alternate routes 


Provide 
flight service 


-Provide clearance 
-Process flight- 
following requests 


Handle 

perturbations 

— Minimize impact of 
hazardous weather 
— Resolve 
non-conformance 
— Respond to 
emergencies 
— Assess equipment 
malfunction 
— Handle special 
operations 


Manage 

resources 


-Manage information 
-Determine impending 
workload 
-Relieve/assume 
control 


Figure 4. En route air traffic controller top-level goal hierarchy 
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Pilots tend to assess the benefit of a route deviation in aircraft-centered terms. They 
evaluate its potential to improve one or more of the five basic elements of the pilot’s objective 
function: safety, legality, schedule, fuel efficiency, and ride quality (Endsley, et al., 1998). 
Against these advantages they weigh the time and effort required of them to request and execute 
the deviation. 

In contrast, controllers tend to assess the benefit of a route deviation in system-centered 
terms. They evaluate its potential to improve their own objective function, which takes into 
account any potential loss of separation, the effect of the deviation on the overall traffic flow (an 
indicator of future workload), and the time and effort required of them and other affected 
controllers to process the deviation (e.g., documentation, coordination, communication). 

So although both pilots and controllers similarly weigh the advantages against the 
workload implications, their re-route assessments in fact have little in common. 

Furthermore, the pilot’s aircraft-centered decision process is isolated from the 
controller’s system-centered decision process because neither agent has ready access to the 
information which the other considers relevant. For example, information regarding schedule, 
fuel efficiency and ride comfort — all of which are important to pilots — is not readily available to 
controllers, so they cannot easily incorporate such considerations into their decision process. 
Furthermore, neither pilots nor controllers have the means to know how much additional 
workload a particular request may create for their counterpart, and consequently it does not 
appear to factor into their decisions. In fact, the additional workload required for them to collect 
such data deters them from seeking it in the first place. 

From this analysis it can be seen that the negotiation of route amendments between pilots 
and controllers is fraught with potential conflicts. With disparate goals, different cost bases, and 
limited information or feedback, it is not surprising that re-route negotiation is a common source 
of conflict between pilots and controllers. 

The information upon which such negotiations are conducted varies, but pilots and 
controllers reported that traffic and weather information often provide the impetus to change path 
and typically impose constraints on the available alternatives. The importance of traffic and 
weather data is of interest, since at present pilots typically have information superiority with 
respect to weather while controllers typically have information superiority with respect to traffic. 
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These results are supported by the experimental observations of Midkiff & Hansman (1992) who 
found that pilots were more assertive towards ATC in weather situations and more compliant in 
traffic situations. In other words, pilots were more assertive when they knew they had an 
information advantage, and they were more compliant when they knew their information to be 
inferior. These observations attest to the power of information and illustrate how information can 
affect negotiation strategy. 

To examine the extent to which information may affect pilot-controller interaction, a 
part-task simulator experiment was undertaken. Given pilots' and controllers' mutual interest in 
the flight path and the different criteria by which they evaluate flight plan deviations, the 
experiment was directed at re-routing situations. The importance of traffic and weather 
information as identified in the comparative analysis was reflected in the experiment's use of 
traffic and weather elements in the scenarios and the availability of a traffic and weather data 
link as the independent variable. 
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Chapter 3 Part-Task Simulator Study 

Based on the findings of the situation awareness analysis, an exploratory experiment was 
conducted to investigate the extent to which shared traffic and weather information may lead 
pilots and air traffic controllers to cooperate or compete when negotiating route amendments. A 
part-task simulator experiment was designed to assess pilot and controller performance and 
behaviors in re-routing situations. Test scenarios focused on tactical routing decisions which 
would preclude the involvement of the Airline Operations Center (AOC). The availability of 
shared traffic and weather information (via digital data link) was manipulated as the independent 
variable in the experiment. Objective and subjective measures of situation awareness, negotiating 
posture, and overall performance were used in combination with experimenters' observations and 
subjects’ comments to assess the overall system effect. 

3.1 Objectives 

The objective of the experiment was to explore the effects of shared information on pilot- 
controller interaction in re-routing situations. Of particular interest were the following: 

• Does the availability of shared information between pilot and controller improve their 
situation awareness? 

• Does the availability of shared information affect pilot and/or controller workload? 

• Does the availability of shared information affect the amount of pilot-controller 
communication? More or fewer calls? Longer or shorter transactions? 

• How does the availability of shared information affect the posture of pilots and 
controllers with regard to re-route negotiation? Do they become more cooperative? Do 
they become more competitive? 

• What is the net effect of shared information on the quality of negotiated route 
amendments and overall traffic flow? What are the benefits? What are the concerns? 
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3.2 Approach 


To explore these issues effectively, the study required a live, realistic and challenging 
environment in which for pilots and controllers to interact. A part-task simulator experiment was 
developed in which two subjects — one pilot and one controller — would interact to handle 
common en route tactical situations in real time. Scenarios were designed to provide enough 
structure to challenge the subjects, but also with enough latitude to allow the subjects to interact 
freely and develop their own options according to their goals and priorities. Scenarios were 
executed alternately with and without a digital data link for the sharing of traffic and weather 
information between the pilot and controller. Comparisons were made both within and between 
subjects. 

3.3 Test Matrix 

The experiment involved six pilot— controller subject pairs. Each subject pair completed 
one of two test matrices. As shown in Table 2, each test matrix contained six test scenarios: three 
scenarios performed with the data link disabled (i.e., no shared information) and three equivalent 
scenarios performed with the data link active (i.e., shared traffic and weather information). The 
only difference between the two test matrices was the status of the data link. In order to fully 
counterbalance the experiment, each successive subject pair performed the opposite test matrix. 
This facilitated between-subject comparisons. 

Table 2. Test matrices 

Test matrix for subjects 1, 3 & 5 Test matrix for subjects 2, 4 & 6 


Scenario 

Number 

Data link 
Disabled 

Data link 
Active 

la 

✓ 


2a 



3a 

✓ 


lb 


/ 

2b 


/ 

3b 


V 


Scenario 

Number 

Data link 
Disabled 

Data link 
Active 

la 


✓ 

2a 



3a 


✓ 

lb 



2b 

✓ 


3b 

S 
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3.4 Independent Variable: Presence of Data Link 

The independent variable for this experiment was the presence of a digital air— ground 
data link which transmitted continuously-updated traffic and weather information between ATC 
and the flight deck. 

The data link was disabled in the baseline configuration. With the data link disabled, 
there was no sharing of information. Figures 5 and 6 illustrate how the weather and traffic 
information was allocated between the pilot and controller. Weather information was available 
only to the subject pilot via his cockpit map display; the subject air traffic controller received no 
weather information. Conversely, traffic information was available only to the subject controller 
via the plan view display; the subject pilot received no traffic information. Information was 
partitioned in this way to establish clear information superiority for one party relative to the 
other. Thus, in the baseline configuration, the pilot was in a superior position with respect to 
weather information but inferior with respect to traffic information. Conversely, the controller 
was in an inferior position with respect to weather information but superior with respect to traffic 
information. 

In the "data link enabled" configuration, weather and traffic information were shared 
between the pilot and controller. Figures 7 and 8 illustrate how the weather and traffic 
information was allocated between the pilot and controller. The baseline weather information 
available to the pilot via the cockpit map display was supplemented with a prototype Cockpit 
Display of Traffic Information (CDTI), as shown in Figure 7. Similarly, the baseline traffic 
information available to the controller via the plan view display was supplemented with a 
prototype graphical weather overlay, as shown in Figure 8. 

The CDTI was based on a prototype by Cashion, et al. (1997) and featured an integrated 
Traffic alert and Collision Avoidance System (TCAS) emulation. Aircraft within 100 n.m. and 
2600 feet of the ownship were shown on the integrated map display. Two considerations were 
given in the design of the CDTI prototype symbology. First, an effort was made to provide the 
pilot with traffic information elements on the CDTI that were equivalent to those available to the 
controller via the aircraft data blocks displayed on the PVD. As shown in Figure 9, information 
elements included ground track, call sign, altitude, ground speed, and climb/descent indication 
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Figure 5. Pilot's display in baseline configuration. 
Note that no traffic information is presented. 


Figure 6. Controller's display in baseline configuration. 
Note that no weather information is presented. The 
cockpit display in Figure 5 belongs to DAL303, shown 
here inside the 6-mile segmented circle, or "J-ball”. 





Figure 7. Pilot's display in "data link enabled" Figure 8. Controller's display in "data link enabled" 

configuration. Note the addition of CDTI symbology. configuration. Note the addition of a weather overlay to the 

plan view display. The cockpit display in Figure 7 belongs 
to DAL303, shown here inside the 6-mile "J-ball”. 




(not shown). The computer identification number in the PVD data block was not 
incorporated as it offers no information of relevance to the pilot. Second, an effort was 
made to present the traffic information in the most useful reference frame for the pilot. 
Since TCAS systems typically present information in an aircraft-relative reference frame, 
relative altitude and relative ground speed were adopted for the CDTI symbology. An 
inverted V symbol indicated the ground track of each aircraft relative to the ownship. A 
data block beside the aircraft symbol denoted the aircraft call sign and relative ground 
speed in knots. Relative altitude was indicated above or below the aircraft (as applicable) 
m hundreds of feet. The CDTI also incorporated TCAS II alerting logic, including traffic 
advisories and resolution advisories. 


Ground 

track 


Cockpit CDTI 



PVD data block 


Call sign 
Ground speed 


Computer 
identification 
number 


Figure 9. Cockpit and PVD traffic symbology 


The pilot's and controller's weather displays provided identical precipitation 
reflectivity imagery based on NEXRAD ground-based weather radar data. The displays 
were capable of depicting seven distinct intensity levels of convective activity in shades 
of green (light intensity, -8 to 0 dbz), amber (moderate intensity, 0 to +8 dbz), and red 
(high intensity, greater than +8 DBZ). Since the radar source was ground-based, the 
weather display had effectively unlimited range and did not suffer from typical airborne 

radar anomalies such as attenuation effects. There was no "tilt" control available to the 
pilot. 


Relative ground speed was computed as the difference in magnitude between the ground speed of the 

intruder and the ownship; a positive value indicated that the intruder’s ground speed exceeded that of the 
ownship. 
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3.5 Test Scenarios 


Test scenarios were designed to probe pilot-controller situation awareness and 
behaviors in re-routing situations. Three basic scenarios were created, each representing 
common en route air traffic situations involving convective weather and moderate- to 
high-density traffic flows. The traffic and weather elements were scripted such that each 
scenario presented the test subjects with two fundamental tasks: a recognition task and a 
negotiation task. 

The recognition task consisted of two situation awareness probes applied in 
parallel: one weather-related and one traffic-related. Both employed the “testable 
response” method (Pritchett & Hansman, 1996; see also Section 3.6.4). Under this 
performance-based methodology, potentially-hazardous traffic and weather conditions 
(one of each) were scripted into each scenario. The hazards were significant enough that 
a test subject who was aware of the hazard(s) would be compelled to respond. Thus, a 
subject’s action or inaction in response to each hazard would provide a binary indication 
of his/her awareness of it. 

The second fundamental task in each scenario was the negotiation task. If the 
subject controller and/or subject pilot recognized one or both of the testable response 
conflicts, their next task was to negotiate an acceptable route amendment to avoid the 
hazard(s). The re-routing decisions were tactical in nature and therefore did not involve 
an AOC. The intent was not to create situations that were necessarily difficult for the 
pilot or controller to resolve. Rather, the intent was to design situations which would play 
on the competing goals of the pilot and controller to offer each subject a fairly obvious — 
yet different — solution, thereby raising the need for re-route negotiation. 
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3.5.1 Airspace Sector 


Test scenarios were set in Indianapolis Center airspace (ZID) in a high-altitude en 
route sector centered at Muncie, Indiana (MIE) (see Figure 10). Sector airspace spanned 
approximately 70 n.m. east-west and 85 n.m. north-south at its widest points, and it 
included altitudes 14,000 feet and above. Neighboring sectors were not depicted for the 
controllers. In situations requiring coordination with a neighboring sector, the subject 
controller was instructed to coordinate with the experiment monitor (a confederate) at 
their side who would accept or reject requests as they were made. 



Figure 10. Indianapolis Center (ZID) high-altitude sector airspace features 
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3.5.2 Traffic Flow 


Traffic flow through the assigned sector was fairly uniform. The traffic mix was 
approximately 15% widebody aircraft, 80% narrowbody aircraft, and 5% 
regional/commuter aircraft. All aircraft transitioned the sector in the cruise phase of 
flight; the subject controller was not faced with any departures or arrivals. Aircraft 
generally adhered to the published airways, except where deviations or direct clearances 
were approved by ATC. The subject controller was provided with a flight strip for each 
aircraft filed to transition his/her sector. The flight strip showed the aircraft's identifying 
information, expected time of arrival at the posting fix, and filed route of flight through 
the sector (see Figure 11). 
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Figure 11. Example en route flight progress strip (with template) 

Traffic densities were high by design in order to make the scenarios challenging, 
given the homogeneity of the aircraft and their routes (e.g., no departures or arrivals). The 
number of aircraft in this relatively small sector averaged about eight aircraft at any given 
time. Scenarios were designed to maintain a regular flow of traffic of between five and 
eleven aircraft. Controllers were given the liberty not to accept an arriving aircraft if the 
sector workload became too hig h. 

Every scenario was designed to have at least one potential traffic conflict. 
Scripted traffic conflicts involved merging traffic only; there were no scripted "blunders" 
or "busted" clearances, although some inadvertent cases did occur. Merging aircraft 
maintained constant airspeed and heading. In some cases, a scripted lateral conflict was 
compounded by an overtaking situation and/or an aircraft climbing to co-altitude. 



3.5.3 Weather Elements 


Every scenario featured one or more weather element. These elements were 
restricted to convective weather patterns: cells and fronts. Winds, icing, and temperature 
profiles were not modeled. The weather elements were static; there was no dynamic 
buildup, dissipation or drift. This simplification was mitigated by the short duration of the 
scenarios (less than ten minutes each). Weather patterns were retrieved from a 
commercial archive of NEXRAD weather data recorded from various sites across the 
continental United States. The elements used in these scenarios ranged from local, low- 
level precipitation to broad, high-intensity frontal systems. 

3.5.4 Scenario Design 

Three basic scenario templates were created for this experiment. The first 
template featured relatively light traffic flow and localized weather. The second template 
featured high traffic flow and a front of moderate-level convective activity. The third 
template featured moderate traffic flow and high-intensity weather. The sections that 
follow provide detail into the features and philosophy of their design. 

3.5.4. 1 Use of Repeated Scenarios 

In order to make within-subject comparisons, it was necessary to perform each 
scenario twice: once without shared information and once with. To reduce the chance that 
a subject would recognize a scenario on its repeated trial, each scenario was modified to 
produce a second scenario which, although superficially distinct, was substantively the 
same. For example, aircraft identifiers were changed (e.g., airline, call sign, transponder 
code) and routes of non-factor aircraft were revised. Routes of confederate aircraft were 
"mirrored" symmetrically about the Muncie fix (MIE) at the center of the sector. In this 
way, while their absolute trajectories were considerably different, their relative 
trajectories remained essentially the same. Where necessary, the altitudes of confederate 
aircraft were amended in keeping with altitude-for-direction conventions. In the 
following sections, the 'a' [and 'b'] version of each numbered scenario represents the 
scenario in its original [and modified] form. 
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3.5.4.2 Scenario 1 


Scenarios la & lb featured relatively light traffic and localized weather, as shown 
in Figures 12 & 13. The subject aircraft was initialized on a dog-leg course through the 
Muncie sector. A traffic conflict was scripted along the subject aircraft's present course. 
Weather obstructed the most likely deviation: the direct path short-cutting the dog leg. 

The traffic conflict was designed such that, if no action were taken by pilot or 
controller, SWA219 [UAL565] would continue climbing to reach Muncie (MIE) co- 
altitude with — and less than a mile ahead of — the subject aircraft, resulting in a loss of 
lateral and vertical separation. The pilot of the confederate aircraft was instructed not to 
change trajectory unless commanded by ATC. Thus, it was incumbent on one subject or 
the other to initiate some action to avoid the conflict. 

Disincentives were used to spoil the trivial solutions. For example, moderate 
turbulence was reported at flight level 330 [350] by the confederate pseudo-pilot of 
DAL881 [DAL214]. This would be a factor if the controller sought to solve the conflict 
by descending one aircraft. Similarly, the aircraft were performance limited to altitudes 
below 40,000 feet, a factor if the controller sought to solve the conflict by climbing one 
aircraft. Finally, the leading aircraft, SWA219 [UAL565], was 35 knots slower than the 
subject aircraft, which would be undesirable to the subject pilot if the controller were to 
sequence the subject aircraft behind the lead aircraft. 

Without the shared information, the optimal solution from the controller’s 
perspective was expected to be a clearance for the subject aircraft direct to FDY [VHP]. 
However, this would conflict with the pilot's goal of avoiding hazardous weather, thereby 
establishing grounds for re-route negotiation. The optimal solution from the pilot’s 
perspective was expected to be the shortest route that would avoid the weather and would 
not require vectors for traffic. 
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Figure 14. Scenario 2a-Subject aircraft (circled at bottom) Figure 15. Scenario 2b-Subject aircraft (circled at right) 
on course to FWA via MIE. on course to VHP via MIE. 



3.5.4.3 Scenario 2 


Scenarios 2a & 2b featured relatively heavy traffic and a moderate-intensity 
weather front with two discernable holes through which traffic was already diverting, as 
shown in Figures 14 & 15. The subject aircraft was bound for FWA [VHP] via MIE and 
was initialized on a course directly into weather at a point between the two holes. 
Conflicting traffic was scripted to occupy the more likely of the two deviations available 
to the subject pilot: the hole representing the shorter deviation for the subject aircraft. 

The traffic flow was designed to favor the longer deviation. The two aircraft 
ahead of the subject aircraft had already opted for the longer deviation. Three aircraft 
merging from the west were scripted to compete with the subject aircraft for the more 
direct hole. As in scenario 1, disincentives were used to spoil the trivial solutions. 
Moderate turbulence was reported at flight level 330 [350] by the confederate pseudo- 
pilot of US A447 [ AAL975] . 

The optimal solution from the controller's perspective was expected to be a 
clearance to follow in trail behind the two aircraft already deviating. The optimal solution 
from the pilot's perspective was expected to be a deviation through the more direct hole 
in the weather. Negotiation of an acceptable route amendment would then ensue. 
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Figure 16. Scenario 3a-Subject aircraft (circled at bottom) Figure 17. Scenario 3b-Subject aircraft (circled at right) 
on course to FWA via MIE. course to VHP via DQN. 




3.5.4.4 Scenario 3 


Scenarios 3a & 3b featured a moderate traffic flow and high-intensity weather 
obstructing one gate into and out of the sector, as shown in Figures 16 & 17. The subject 
aircraft was bound for FWA via MIE [VHP via DQN] and was initialized on a course 
directly into weather at a range of approximately 60 n.m. Conflicting traffic was scripted 
to compete for the shortest path around the weather. 

The optimal solution from the controller’s perspective was expected to be a 
clearance to remain at flight level 330 [descend to flight level 310], below the merging 
traffic at flight level 370 [350], However, this solution would be scuttled by a report of 
moderate turbulence at flight level 330 [310] by the confederate pseudo-pilot of SWA 155 
[COA329]. The optimal solution from the pilot’s perspective was expected to be a climb 
to flight level 370 [remain at flight level 350] and a sequence position ahead of the 
merging traffic, NWA708 [USA512]. Negotiation of an acceptable route amendment 
would then follow. 

3.6 Protocol 
3.6.1 Test Facility 

The part-task simulator study was conducted in the Distributed Air Traffic 
Simulation Facility (Amonlirdviman, et al., 1998) located at the MIT International Center 
for Air Transportation. This facility provided a virtual airspace environment capable of 
hosting multiple flight simulators, ATC simulators, and pseudo-aircraft simulators in a 
single, interactive, real-time simulation. For this experiment, the facility was configured 
with one advanced cockpit simulator, one ATC simulator, and one pseudo-aircraft 
simulator (see Figure 18). Live voice and data communication was provided between 
each simulator and the simulation host. In order to minimize non-radio interaction 
between the two subjects, the cockpit simulator and the ATC simulator were physically 
separated as shown in Figure 19. 
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Figure 18. Distributed simulator facility configuration 
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Figure 19. Distributed air traffic simulation facility 
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3.6.1.1 Advanced Cockpit Simulator (ACS) 

The advanced cockpit simulator was a part-task flight simulator (see Figure 20). 
Cockpit hardware included an electronic display emulation, a Mode Control Panel 
(MCP), a side-stick controller, a Control and Display Unit (CDU), and a center pedestal 
housing the flap lever, speed brake lever, and throttle quadrant. 



Figure 20. Advanced cockpit simulator 

The primary components of the electronic displays were the Primary Flight 
Display (PFD) and the Electronic Horizontal Situation Indicator (EHSI). The PFD, as 
illustrated in Figure 21, emulated the PFD symbology and layout used in the Boeing 
747-400 Electronic Flight Instrument System (EFIS). The EHSI was an integrated display 
prototype, with the capability to display route, weather, and traffic information, as shown 
in Figure 22. 
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Figure 21. Primary Flight Display (PFD) for the 
Advanced Cockpit Simulator (ACS) 


Figure 22. Electronic Horizontal Situation Indicator (EHSI) 
with prototype weather and traffic displays 


Figure 23. ACS mode control panel 

Subject pilots were free to operate their aircraft with whatever level of automation 
they deemed appropriate. Autopilot modes and target states were controlled via the mode 
control panel shown in Figure 23. The MCP was authentic 737-200 hardware and 
featured modes for heading hold, heading select, altitude hold, vertical speed, level 
change, lateral navigation (LNAV), and vertical navigation (VNAV). The aircraft 
trajectory in LNAV and VNAV modes was governed by a Flight Management System 
(FMS) emulation. The FMS enabled the pilot to 
input complex lateral and vertical flight plan 
segments into the autoflight system via the CDU. 

The CDU faceplate (see Figure 24) was actual 
Boeing 757/767 hardware. A side-stick controller 
was available for manual control if the subject 
pilot chose to disconnect the autopilot. Range of 
the map display could be increased or decreased 
by pushing buttons on the instrument panel using 
a mouse. Minimum range was five nautical 
miles; maximum range was not limited. The pilot 
was outfitted with a headset to communicate with 

ATC and other aircraft. 

Figure 24. Control and Display Unit 

3.6.1.2 Air Traffic Control Simulator 

The air traffic control part-task simulator was modeled after the Ml consoles used 
at the 20 Air Route Traffic Control Centers (ARTCC) in the United States. The ATC 
simulator was comprised of the Plan View Display (PVD), Computer Readout Display 
(CRD), and Data Entry Control (DEC) system, and also included a map of the sector and 




a set of flight strips for the aircraft currently in or approaching the sector (see Figure 25). 
The PVD displayed radar tracks and full data blocks for all tracked aircraft in the 
simulation within its assigned airspace sector, along with sector adaptation data such as 
airports, navigation aids, and airways. Although aircraft position updates were received 
continuously from the simulation host, target positions were updated only once every 12 
seconds on the PVD to emulate the update rate of actual en route ATC equipment. 
Slewball inputs and/or alphanumeric keyboard commands were used to display 
supplementary information such as a target's current trajectory, filed flight plan, or 
position history. The same input devices were used to zoom or offset the plan view 
display. All data entry keyboard/mouse input sequences emulated those of the real DEC. 
In addition, the NEXRAD-based weather overlay prototype was integrated into the ATC 
display. Figure 26 shows the ATC simulator plan view display. The controller was 
outfitted with a headset to communicate with the aircraft transitioning the sector. 



Figure 25. Air traffic controller's workstation 
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Figure 26. En route ATC Plan View Display (PVD)— Flight plan information 
displayed on the CRD and the 6-mile segmented circle (a.k.a. "J-baU M ) are for the 
subject aircraft being simulated by the ACS. 

3.6.1.3 Pseudo-Aircraft Simulator and Control Station 

All confederate air traffic (i.e., all aircraft not flown by the subject pilot) were 
simulated and piloted from the pseudo-aircraft simulator and control station shown in 
Figure 27. This SGI Octane-based application provided several key functions. For each 
pseudo-aircraft in the scenario, it simulated the aircraft dynamics and an on-board flight 
management function. In this way, each pseudo-aircraft could be preprogrammed with its 
own unique four-dimensional flight plan. 



Figure 27. Pseudo-aircraft simulator and control station 


The software also featured a graphical user interface to enable real-time control of 
pseudo-aircraft trajectories. Through an intuitive "point-and-click" interface, the pseudo- 
pilot was able to exercise outer-loop control of each aircraft’s autoflight system. As 
shown in Figure 28 , scenario feedback was provided to the pseudo-pilot via a large plan- 
view window depicting all of the air traffic in a region. When the pseudo-pilot selected a 
specific aircraft (by clicking on it with the mouse button), state feedback for the selected 
aircraft was provided via a "pseudo-cockpit" window (shown to the left of the plan-view 
window) which indicated the aircraft’s current attitude, airspeed, altitude, heading, and 
flight control modes. 
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The plan- view window and the pseudo-cockpit window also functioned as input 
devices for autoflight mode transitions and state commands. For example, the altitude 
tape served as a flight level change input. When the mouse pointer was positioned on the 
altitude tape, a command bug would appear. Moving the mouse pointer up the tape would 
increase the armed altitude; moving the mouse pointer down the tape would decrease the 
armed altitude. Clicking the second mouse button would activate the altitude command. 
The autoflight system would transition from VNAV to FLCH, and the selected aircraft 
would initiate a climb or descent accordingly. The pitch attitude indicator would go up or 
down accordingly, thereby providing immediate feedback that the desired input had been 
invoked. Analogous controls were implemented for speed and heading. Autoflight mode 
transitions could be manually enacted by clicking on the flight mode annunciator located 
above the attitude indicator. 
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3.6.1.4 Scenario Design Tool 


Scenario designs were implemented with the use of the script development tools 
which were an integrated part of the pseudo-aircraft simulation and control station. The 
scenario design tools included some elements of a robust situation generation approach 
which enabled the user to build and store a unique flight plan of four-dimensional 
waypoints for each pseudo-aircraft in the scenario (Johnson, 1995). In playback mode, 
basic FMS-type navigation functions were implemented to control each pseudo-aircraft 
along its four-dimensional flight plan. This capability enabled the user to iteratively build 
and test air traffic scenarios off-line featuring many 3 aircraft of various type, each having 
complex lateral, vertical and speed profiles. The scenario design tools use the same 
graphical user interface as the pseudo-aircraft simulator and control station, as shown in 
Figure 28. Performance constraints such as maximum speed, climb, and rate of descent 
were enforced to ensure reasonable pseudo-aircraft trajectories for the given aircraft type. 

3.6.2 Subjects 

Six air traffic control specialists and six commercial pilots were recruited to 
participate in this study. All participants were volunteers. 

All six air traffic control specialists were Full Performance Level (FPL) 
controllers currently on staff at an Air Route Traffic Control Center (ARTCC). Radar 
experience ranged from 7 to 20 years, with a mean of 13.3 years. The controllers were 
between 3 1 and 48 years of age, with a mean of 38.0. 

All six pilots were active jet aircraft pilots with an Air Transport Pilot (ATP) 
rating. Flight experience ranged from 6000 to 16,000 hours, with a mean of 10,1 17 hours. 
All had experience in “glass cockpit” and FMS-equipped airplanes. Pilots were between 
40 and 53 years of age, with a mean of 45.2. Two of the pilots were corporate pilots and 
four were pilots with major airlines. 


3 There is no inherent limit to the number of aircraft that can be scripted for a given scenario. While there 
are likely to be practical limitations due to computer memory and/or network bandwidth, simulations of 
100 aircraft have been demonstrated with no adverse effects. 
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3.6.3 Procedure 


The experimental procedure included three basic activities: preliminary briefings 
and training, formal testing, and final debriefing. Completion of the entire protocol 
typically required four hours. 

3.6.3. 1 Preliminary Briefings and Training 

To begin, the pilot subject and the controller subject were briefed separately on 
the overall agenda for the session and their roles and responsibilities as subjects. Initial 
briefing materials are included in Appendix C. 

Following the initial briefing, the subject pilot and controller were brought to their 
respective simulators where they were briefed on the airspace environment, operation of 
the simulator, special procedures, and their required tasks. There was no interaction 
between the two subjects during this training phase. The training checklists are included 
in Appendix D. During the pilot briefing, the features of the cockpit displays, flight 
control computers, and voice communication system were explained in detail. It was 
impressed upon them that their flight was straining to remain on schedule, that they were 
carrying revenue passengers, and that ride comfort was therefore a consideration. During 
the controller briefing, features of the Plan View Display, Data Entry Control (DEC) 
system, flight strips and voice communication system were covered. The controllers were 
reminded that current air traffic control procedures and standards were in force. Each 
subject was given time off-line to become familiar with the controls, displays, and 
simulation environment before engaging in any interactive, multi-agent simulation 
exercises. 

Once both subjects were comfortable with the operation of their respective 
simulators, an interactive practice scenario was conducted to allow the subjects to operate 
their simulators and use the radio in a live, interactive, ffee-play scenario representative 
of the formal test environment. Practice scenarios were repeated until both subjects were 
confident in their ability to operate the equipment and comfortable with their assigned 
roles, responsibilities, and the simulation environment in general. 
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3.6.3.2 Formal Testing 

Formal testing began immediately after training was concluded. Each formal test 
scenario began with the simulation "frozen". The subject pilot's aircraft was initialized in 
cruise trim and on course with the autoflight system engaged and tracking its 
preprogrammed route (LNAV). Both subjects were given a five-minute period in which 
to survey their static situation as shown on their respective simulator displays. Controllers 
were allowed to organize and annotate their flight strips to develop a "picture" of the 
traffic in and about their sector. Pilots were allowed to review their flight plan and the 
local weather as portrayed on their map display. The subjects were told in advance 
whether the air— ground data link would or would not be active for the queued scenario. 
This was intended to establish a priori an understanding of their relative information 
superiority (or inferiority) as a basis for any subsequent negotiation. No suggestions were 
given as to how they should make use of the available information or how to exploit any 
information advantage they might have. For cases in which the data link was disabled, 
controllers were notified of convective weather activity in the area, but the specific 
location and intensity of the weather was not specified. Similarly, pilots were notified of 
traffic in the area, but the specific location and altitude of the traffic was not specified. 
Except for the data link status (data link enabled or disabled), the simulator setup was 
identical for each experimental run. 

Following the situation assessment period, the simulation was started. Each 
scenario began with a number of scripted radio calls from one or more of the confederate 
pseudo-aircraft. The subject pilot was instructed to check in with Indy Center at his first 
convenience as though he had just been handed off by the previous ATC sector. The 
subject pilot and subject controller were then free to take whatever action or contact 
whichever person they deemed necessary to accomplish their goals within the bounds of 
their assigned roles and responsibilities. Subjects were encouraged to verbalize (off the 
frequency) their thoughts, observations, and decision processes during the scenario as 
much as possible. An observer was assigned to each subject to record these comments 
and other noteworthy actions. Each scenario was allowed to run for approximately ten 
minutes, enough time for the subject aircraft to transition the airspace sector. 
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Following each scenario, each subject was administered a workload survey. In 
addition, following all scenarios in which the data link was disabled, controller subjects 
were asked to indicate on a sector map their best estimate of the location of the 
convective weather. These and other data collection activities are discussed in Section 
3.6.4. 

3.6.3.3 Debriefing 

Upon completion of the six-scenario protocol, test subjects were interviewed 
individually and then jointly to elicit from them the advantages, disadvantages, and issues 
regarding the shared traffic and weather information. Subjective evaluations were 
solicited via a brief questionnaire. 

3.6.4 Metrics and Data Analysis 

The objective of this exploratory study was to identify changes in pilot-controller 
interaction with the availability of shared information. Five types of data were collected 
to help characterize and quantify changes in their interaction and overall performance: 

• situation awareness data 

• aircraft trajectory data 

• voice data 

• workload data 

• subjective ratings. 

Pilot and controller situation awareness was measured using the performance- 
based "testable response" method (Pritchett & Hansman, 1996). In each scenario, subjects 
were presented with one weather- and one traffic-related testable response condition (e.g., 
a storm cell, a converging aircraft). The conflicts were designed such that, if a subject had 
sufficient situation awareness, a deliberate action was required. A subject’s action or 
inaction in response to the weather or traffic conflict provided a measurable indication of 
that subject's situation awareness with respect to the specific weather or traffic conflict, 
respectively. Cases in which one subject’s response to a conflict prompted or precluded 
the other subject’s response were not considered valid data points. 
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In the absence of reliable weather information, controllers use pilot reports 
(PIREPs), aircraft trajectories, pilot requests, and other clues to construct a mental picture 
of the areas affected by weather and to project how the traffic flow will be affected. To 
gain some insight into the accuracy of this heuristic, for scenarios in which a weather 
overlay was not provided, controllers were asked to draw on a sector map the location of 
any weather cells as inferred from the aircraft trajectories and information attained over 
the voice channel. Drawings were made at the conclusion of each scenario, and the 
controllers were allowed to refer to the PVD (frozen at the end of the scenario run) as 
necessary. The drawings were subsequently compared against the actual location of the 
weather to assess the degree to which they coincided. 

Aircraft trajectory data were recorded for every aircraft in the simulation (i.e., the 
subject aircraft and all pseudo-aircraft). The trajectory data were recorded at a rate of 
approximately 10 Hz and included the parameters listed in Table 3. 


Table 3. Recorded trajectory parameters 


Aircraft call sign and type Indicated airspeed 
Transponder code True airspeed 

Latitude Ground speed 

Longitude Vertical speed 

Altitude Pitch, roll, heading 

Radio altitude Flight path angle 


Commanded altitude 
Commanded airspeed 
Commanded heading 
Commanded vertical speed 
Flight plan data 


Using these recorded data, it was possible to reconstruct and replay each scenario. 
The reconstructed flight paths were used to assess the strategies employed by each 
subject and to identify separation violations and other events. 

Voice data were recorded and used to characterize the relative cooperativeness/ 
competitiveness of pilot— controller interactions and to provide insight into changes in 
each subject's strategies. All voice transmissions between the subject pilot, the subject 
controller, and the pseudo-pilot were digitally recorded. The recordings were transcribed 
verbatim to written copy. For each transmission, the transcript identified the speaker and 
provided a time stamp. Using the methodology of Prinzo, Britton, & Hendrix (1995), 
each transmission was divided into discrete speech acts, and each speech act was coded 
by category and topic. An example of the speech act coding sheet is provided in 
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Appendix E. Speech acts were assigned to one of the thirteen speech act categories listed 
in Table 5, adapted from Foushee, Lauber, Baetge, & Acomb (1986) to accommodate 
pilot-controller communications. Each speech act was also identified with its applicable 
topic(s), if any. Eight speech act topics were defined, as listed in Table 4. 

Table 4. Speech act topics 

Route / Heading Weather 

Altitude Traffic 

Speed Ride / Turbulence 

Radio frequency Other 

The coded voice transcript data were tallied, and paired t-tests were applied to assess the 
significance of the differences between the means with and without the shared data link. 

In addition to the "radio" voice data, the off-frequency comments of the pilots and 
controllers regarding their strategic considerations, options and tradeoffs were also 
recorded. These comments were reviewed for indications of strategic differences in their 
route planning when shared information was available, and for any shift in their attitude 
toward their counterpart and/or the negotiated re-routing. 

Workload data were collected and analyzed in accordance with the NASA Task 
Load Index (NASA-TLX) methodology (Hart & Staveland, 1988). Following each test 
scenario, each subject completed a brief "workload rating" survey; at the end of all 
scenarios, each subject completed the "workload sources" survey. A sample of each 
survey in provided in Appendix F. Survey responses were used to compute a NASA-TLX 
composite workload score for each test scenario for each test subject. Paired t-tests were 
applied to assess the statistical significance of the data. 


At the conclusion of the experiment, each subject was asked to provide a 
subjective rating as to the value of the shared traffic and weather information in 
performing their job functions. An example of the rating format is shown in Figure 29. 



very somewhat neutral somewhat very 

detrimental detrimental valuable valuable 

Figure 29. Subjective rating format 
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Table 5. Speech act categories (adapted from Foushee, etal. (1986)) 


Command 

A specific assignment of responsibility by one group 
member to another. 

Request 

A request for another party to take an action, or 
a request for permission to take an action. 

Acknowledgement 

a) Makes known that a prior speech act was heard; 

b) Does not supply additional information; 

c) Does not evaluate a previous speech act. 

Courtesy 

Word(s) or phrase(s) spoken as an act of courtesy. 

Advisory 

Recognizing and/or noting a fact or occurrence 
relating to the task. 

Suggestion 

Recommendation for some specific course of action. 

Inquiry 

A request for factual information relating to the task. 
Not a request for action. 

Answer 

Speech act supplying information beyond mere 
agreement, disagreement, or acknowledgment. 

Statement of intent 

Announcement of an intended action by speaker. 
Includes statements referring to present and future 
actions, but not to previous actions. 

Response 

uncertainty 

Statement indicating uncertainty or lack of 
information with which to respond to a speech act. 

Embarrassment 

Any comment apologizing for an incorrect response, 
etc. 

Repeat 

Restatement of a previous speech act. 

Non-codable 

Speech act which is unintelligible or unclassifiable 
with respect to the present coding scheme. 
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Chapter 4 Results 

This chapter presents the experimental results. First, the effect of shared 
information on pilot and controller situation awareness is assessed. This includes a 
discussion of controllers' ability to build good situation awareness regarding the effect of 
weather on traffic flows in the absence of graphical weather information. Section 4.2 
investigates the separation violations that occurred over the course of the experiment to 
assess the relationship between shared information and performance. This is followed by 
the presentation of the communication analysis, addressing the effect of shared 
information on pilot-controller interactions. Lastly, the workload data is presented, and 
the pilots’ and controllers’ subjective ratings of the value of shared information are 
discussed. 

4.1 Situation Awareness 

Two approaches were taken to the situation awareness (S A) analysis. The first 
approach used the performance-based testable response methodology to assess the 
situation awareness of the subject pilot and subject controller with respect to weather and 
traffic in real time. The second approach to assessing situation awareness used a visual 
recall task at the conclusion of each scenario to focus on the situation awareness of the 
subject controllers with respect to weather only. The two methodologies are discussed in 
greater detail in Section 3.6.4. The two sets of results are presented below. 

4.1.1 Testable Response Data 

Each test scenario included one weather-related testable response condition and 
one traffic-related testable response condition. Both the pilot and controller were 
monitored for their awareness of each testable response condition. 
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Traffic SA 

Without datalink With datalink 



Figure 30. Pilot and controller 
awareness of traffic-related testable 
response conditions 


Weather SA 


Without datalink With datalink 



Figure 31. Pilot and controller 
awareness of weather-related testable 
response conditions 


Figure 30 summarizes the results of the traffic-related testable response probes. 
Pilots, without the benefit of a traffic display in the baseline configuration, did not 
demonstrate awareness of any of the traffic-related testable response conditions. When 
provided a shared traffic display, pilots demonstrated awareness of 56% of the traffic- 
related testable response conditions. In many cases, the controller recognized the traffic 
conflict before it became a significant threat to the pilot and either advised the pilot of the 
traffic or vectored the pilot accordingly. In such cases, the pilot's opportunity to 
independently recognize and respond to the hazard was precluded, and the testable 
response result for the pilot therefore was labeled "ambiguous". 

Controllers, having the benefit of their plan view traffic display for all test 
scenarios, demonstrated a high level of awareness of the traffic-related testable response 
conditions. In some cases, a deviation requested by the subject pilot resolved the traffic- 
related testable response condition before it arose; such cases were labeled "ambiguous" 
with respect to controller situation awareness. 

Figure 31 summarizes the results of the weather-related testable response probes. 
Pilots, having the benefit of the weather display for all test scenarios, demonstrated 
awareness of all of the weather-related testable response conditions. Controllers, without 
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the benefit of a weather display in the baseline configuration, demonstrated awareness of 
only 50% of the weather-related testable response conditions. When provided a shared 
weather display, controllers demonstrated awareness of 94% of the weather-related 
testable response conditions. In one case, a controller gave conflicting indications of 
his/her awareness of the weather conditions. For that case, the controller's testable 
response result was labeled "ambiguous". 

These results indicate that pilot situation awareness with respect to traffic 
improved with the addition of a CDTI. Similarly, the results suggest that controller 
situation awareness with respect to weather improved with the addition of a weather 
overlay to their plan view display. These results confirm that shared information via air- 
ground data link can improve situation awareness for both pilots and controllers. 

4.1.2 Controllers’ Weather Awareness Data 

The benefit derived by the controllers from the prototype weather display is made 
clearer when one compares the NEXRAD image of the weather situation as provided by 
the data link against the mental image of the weather situation as constructed by each 
controller in the absence of the data link. The drawings in Figures 32 through 37 facilitate 
this comparison. They illustrate the degree to which each controller was able to build an 
accurate mental model of the weather situation for a given scenario. The actual location 
of the weather is overlaid for reference. The nine figures on pages 50-51 correspond to 
subjects 1, 3 and 5 and are arranged in a matrix as follows: each row is subject-specific, 
containing the three drawings performed by each subject; each column is scenario- 
specific, containing each subject's drawings for the three non-datalinked scenarios. By 
comparing rows and columns separately, one can assess whether the dominating factor in 
developing weather situation awareness is the skill level of the controller or the nature of 
the weather pattern and traffic flows. The nine figures on pages 52-53 are similarly 
arranged and correspond to the weather drawings of subjects 2, 4 and 6. (Recall from 
Table 2 that the non-datalinked test cases for subjects 1, 3 and 5 were scenarios la, 2a 
and 3a, while the non-datalinked test cases for subjects 2, 4 and 6 were scenarios lb, 2b 
and 3b.) 
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Figure 32. ATC weather recall results 
(Scenario la) 
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Figure 35. ATC weather recall results 
(Scenario lb) 








(a) Controller 2 (a) Controller 2 



Figure 36. ATC weather recall results Figure 37. ATC weather recall results 

(Scenario 2b) (Scenario 3b) 





A metric was developed to lend some quantitative insight into the controllers’ 
performance on this exercise. The airways within the sector airspace were divided into 26 
airway segments of between 15 and 20 n.m. each. For each scenario, the 26 airway 
segments were categorized as either “weather-obstructed” or “clear”. For this simple 
metric, any airway segment that intersected the perimeter of a weather cell or lay 
completely within a weather cell was considered to be “weather-obstructed”. The 
intensity of the cell (i.e., the color of the NEXRAD image) was not considered. All 
segments that were not “weather-obstructed” were considered to be “clear”. Recall that 
the weather image was static, so it was not possible for an airway segment to change 
from weather-obstructed to clear, or vice versa. 

Having categorized the actual coverage of the weather in terms of airway 
segments, the same procedure was applied to the controller sketches presented in the 
figures above (without the weather overlay). Airway segments that intersected a region 
drawn by the controller were recorded as “weather-obstructed”. The remaining airway 
segments were recorded as “clear”. 

Figure 38 compares the controllers’ inferred understanding of which airways were 
weather-impacted against the actual list of weather-impacted airways. The dark-colored 
portions of the pie graph represent the number of airway segments which were obstructed 
by weather. Of those airway segments, the controller subjects correctly identified fewer 
than half. Conversely, the light-colored portions of the pie graph represent the number of 
airway segments that did not intercept weather. Of those 333 airway segments, the 
controller subjects mistakenly identified 46 of them (14%) as being impacted by weather. 



sector airways 
obstructed by weather 
(135) 


sector airways not 
obstructed by weather 
(333) 



airways correctly identified 
as obstructed (62) 

airways incorrectly 
identified as clear (73) 

airways correctly 
identified as clear (287) 

airways incorrectly identified 
as obstructed (46) 


Figure 38. Cumulative weather inference results 
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In order to understand how these misperceptions develop, consider the figures on 
pages 56 through 59. These figures are identical to those presented on pages 50 through 
53, except that the trajectories of all of the aircraft in the sector during the scenario have 
been overlaid. Note that, to first order, the trajectories tend to wind around the regions 
drawn by the controllers. This is consistent with the strategy controllers report using to 
deal with weather disturbances. In the absence of displayed weather, controllers attempt 
to identify and bound the weather-impacted areas in their sector, and they mentally set 
those boundaries based on the trajectories of the aircraft they control as best they can 
recall them. As a memory aid, some controllers use grease pencils to literally draw these 
bounded weather regions on the PVD screen, revising the boundaries where necessary as 
dictated by the most recent aircraft trajectories and pilot reports. 

Examining the trajectory data, note that the regions drawn by the controllers 
closely reflect the curved trajectories of aircraft who negotiated course deviations with 
ATC. Note, too, that there are several cases in which a controller drew a weather region 
over an area which had clearly been traversed by one or more aircraft. (Figures 41 (a) and 
(c) contain prime examples.) The most flagrant cases involve aircraft which made no 
course deviations and, in general, did not have radio contact with the controller other than 
on their arrival to and departure from the sector. This suggests that during periods of high 
workload the trajectories of non-routine, deviating aircraft may figure more prominently 
in the minds of controllers as they attempt to build and maintain a mental picture of the 
weather situation. 
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Figure 39. ATC weather recall results 
with tracks overlaid (Scenario la) 










Figure 40. ATC weather recall results Figure 41. ATC weather recall results 

with tracks overlaid (Scenario 2a) with tracks overlaid (Scenario 3a) 
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Figure 43. ATC weather recall results 
with tracks overlaid (Scenario 2b) 


Figure 44. ATC weather recall results 
with tracks overlaid (Scenario 3b) 














In analyzing this data, it must be acknowledged that the recall task that the 
controllers were asked to perform for this analysis is subtly different than the task they 
typically perform when adverse weather conditions arise. The exercise for this 
experiment was a post-test task; data was not collected in real-time. In addition, the 
weather resources made available to the controllers were limited: the PVD did not feature 
the usual (albeit rudimentary) weather symbology; the controllers did not have access to 
an advanced weather display or weather briefing, which is typically a few feet away at 
the supervisor’s desk; and, the controllers were not permitted to use the grease pencil 
method. 

4.2 Separation Violations 


In the 36 test scenarios, five separation violations were observed, all of which 
occurred with the data link disabled. A loss of separation was defined in accordance with 
en route ATC standards: lateral separation of less than five nautical miles and vertical 
separation of less than 1000 feet. Figure 45 indicates the closest points of approach for 
the five separation violations. The upper right comer corresponds to the 5-nm, 1000-foot 
separation standard which defines a loss of separation for en route operations. The lower 
left comer corresponds to zero separation — a collision. 
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Lateral Separation (nm) 


1 Conflict precipitated by a late 
deviation around weather 

2 Several aircraft deviating 
through same hole in the 
weather 

3 Delayed execution of handoff; 
conflict occurred outside the 
sector 

4 Pilot blunder, detected by ATC 
(turned in wrong direction) 

5 Pilot blunder, detected by pilot 
(wrong A/P mode for descent) 


Figure 45. Closest points of approach for the five separation violations 
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It is important to note that several factors made the controllers’ tasks in these test 
scenarios unusually demanding. First, the test scenarios were challenging by design. The 
sector’s small size coupled with higher-than-typical traffic densities increased the tempo 
of activity in the sector and shortened the planning timeframe from strategic to tactical. 
Furthermore, controllers were operating an air traffic sector other than their usual "home" 
sector and did not have the benefit of a conflict alert function or a D-side controller to 
assist them. 

The five separation violations fall into two general categories. As will be 
discussed, events #1 and #2 were serious near-miss incidents which appear to be 
attributable to poor situation awareness, in this case the byproduct of severe weather and 
traffic constraints. Events #3, #4 and #5 were borderline cases attributable to high 
workload and distraction on the part of the controller, pilot or pseudo-pilot. 

Events #1 and #2 occurred as several aircraft were attempting to deviate through a 
corridor in a weather front. In each case, with no weather information available, the 
controllers had difficulty anticipating deviation requests and developing a coherent flow 
strategy. As a result, they had to react to several urgent requests in a short time period. 
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Figure 46. Loss of separation #1 (CPA: <100 ft & 0 ft.) 

A snapshot of event #1 is provided in Figure 46. UAL323 was descended from 
FL370 to FL350 near RID to separate conflicting traffic. As the scenario developed, four 
aircraft requested clearance through the same hole in the weather. In attempting to 
accommodate all of their requests, the controller apparently lost awareness that two 
aircraft were co-altitude and in opposite directions through the hole. UAL323 and UAL 
751 eventually closed to within 100 feet. The controller recognized the situation after the 
two aircraft had passed. 
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Figure 47. Loss of separation #2 (CPA: 0.5 n.m. & 0 ft.) 

A snapshot of event #2 is provided in Figure 47. DAL 189 was descended from 
FL370 to FL350 southwest of MIE to make way for other traffic coming through the 
hole. DAL768 requested and received a last-minute clearance to deviate 15 miles to the 
left of its FWA-VHP course. These two aircraft eventually came within one half mile of 
one another. The controller recognized the conflict after the two aircraft had passed. 
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Figure 48. Loss of separation #3 (CPA: 4.5 n.m. & 0 ft.) 


Event #3 occurred outside the sector boundary between an incoming aircraft and 
an outgoing aircraft, as shown in Figure 48. At the time of the encounter, the outgoing 
aircraft was under the control of the subject controller, but the incoming aircraft was not. 
The incoming and outgoing aircraft were both level at FL350 on headings of 190 degrees 
and 340 degrees, respectively (just within the 180-to-359 degree heading-for-altitude 
standard). While responding to a request from DAL831 (the subject aircraft) for a ride 
report, the subject controller recognized the impending conflict at FWA and issued 
avoidance instructions to both COA636 and NWA847. 
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Events #4 and #5 occurred as a result of pilot blunders. In event #4 (see Figure 
49), the pseudo-pilot inadvertently made a right turn into traffic instead of a left turn 
away from traffic as commanded by ATC. The controller recognized the blunder before 
the pilot and issued instructions to resolve the situation. In event #5 (see Figure 50), the 
subject pilot selected an inappropriate autopilot mode midway into a crossing descent. 
This resulted in an unintended 500-foot climb before the pilot was able to recognize and 
correct his mistake. The interruption in the descent profile resulted in a loss of separation. 
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ATC:“Turn 20 degrees left, 
vectors traffic.” 

(Aircraft turns 20 deg right.) 


ATC: “Did you turn 20 left or 
right? Turn 40 degrees left 
now sir, vectors traffic.” 


Figure 49. Loss of separation #4 (GPA: 5 n.m. & 750 ft) 



corrects mistake 

Figure 50. Loss of separation #5 (CPA: 4.5 n.m. & 1000 ft.) 
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The fact that all of the separation violations occurred in the non-datalinked 
environment suggests that shared information may help controllers build and maintain 
situation awareness with regard to separation issues. In events #1 and #2, it appears that 
controllers did not have sufficient situation awareness to adequately anticipate and plan 
for the disturbances in the traffic flow brought about by the severe weather constraints. In 
events #3, #4 and #5, high workload in one part of the sector appears to have caused the 
controller to be less vigilant with regard to handoff status and aircraft conformance in 
another part of the sector. 

4.3 Communication and Negotiation 

All radio communication was recorded, coded by category and topic, and 
analyzed. Figure 51 illustrates how the transactions conducted over the voice channel 
changed with the introduction of the data link. As shown at the left, the number of 
transactions between the pilot and controller decreased slightly when the data link was 
introduced. Despite this decrease, the number of transactions for negotiating re-route 
clearances remained virtually constant, and the number of other transactions (including 
traffic advisories, ride reports, etc.) decreased. These results are not statistically 
significant, however. 

Figures 52 and 53 illustrate how the character of pilot— controller interaction 
changed when the data link was introduced. Figure 52 shows that requests by the subject 
pilot and commands by the subject controller (to all aircraft) both dropped slightly, albeit 
not significantly. With the data link enabled, the subject pilot and subject controller made 
more voluntary suggestions to one another for specific route amendments. For example, 
consider the following exchange from scenario 2b (refer to Figure 15): 

AAL303: INDY CENTER AMERICAN THREE OH THREE, FLIGHT LEVEL 

THREE NINE ZERO, LIKE TO DEVIATE HEADING ABOUT TWO 
FIFTEEN FOR ABOUT FORTY MILES FOR WEATHER. 

ATC: AMERICAN THREE ZERO THREE, ROGER. I SHOW A BREAK IN 

THE WEATHER THAT'S ABOUT YOUR ONE O'CLOCK. HAVE YOU 
CONSIDERED A DEVIATE ABOUT TEN TO THE RIGHT awn THEN 
DIRECT INDY? 
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Re-route Other 

negotiations transactions 


Figure 51. Voice communication transactions by topic 



Figure 52. Voice transmissions by category 



Weather 
inquiries from 
ATC 


Weather 
advisories by 
ATC* 


Figure 53. ATC voice transmissions regarding weather 


indicates a statistically significant result. 
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The deviation suggested by the controller was a more direct path than the pilot’s 
requested deviation, saving the pilot approximately four minutes’ flying time. There was 
no apparent benefit to the controller other than the satisfaction of having provided 
improved service. Furthermore, the controller appeared to incur additional workload, as 
the suggested deviation required careful sequencing with merging traffic from the north. 
This kind of verbal exchange of re-routing ideas, options and preferences was rarely 
evident when the data link was disabled. This result is marginally statistically significant 
at the 9% level (p < 0.09). In addition. Figure 53 illustrates that controllers were more 
proactive in providing weather advisories to pilots when they had the weather 
information overlay. This result is statistically significant at the 1% level (p < 0.01). 

Together these results are indicative of more cooperative interaction between pilots and 
controllers. 


4.4 Workload 


Pilot and controller workload was measured using NASA-TLX. The results 
exhibited high variance, both between subjects and within subjects, as indicated by the 
wide error bars in Figures 54 & 55. In general, the availability of shared information did 
not appear to affect pilot or controller workload in any systemic way. 
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Figure 54. Pilot composite workload 


Figure 55. Controller composite workload 



4.5 Subjective Responses 

At the conclusion of each test session, subjects were asked to provide a subjective 
rating of the value of the shared information on a scale ranging from "very detrimental" 
to "neutral" to "very valuable”. Table 6 summarizes the responses of pilots and 
controllers. Pilot feedback was unanimously favorable, and all of the controllers rated the 
information as "very valuable". 

Table 6. Subjective ratings of the value of shared information 



Pilots 

Controllers 

Very valuable 

Somewhat valuable 
Neutral 

Somewhat detrimental 
Very detrimental 

✓ 

✓ ✓✓ 

✓✓✓✓✓✓ 


While controllers were enthusiastic in their support for the shared weather 
display, their opinions on sharing their traffic information with the cockpit were mixed. 
Some controllers suggested that it could be useful to controllers and pilots when 
sequencing aircraft in the terminal area. Others expressed concern that arming pilots with 
such information might make pilots "less complacent" with regard to their approved 
clearances or assigned vectors. During the course of this experiment, pilot-controller 
exchanges were observed that validate each of these opinions. 

4.6 Discussion 

It was anticipated that the sharing of information would change the balance of 
information and, given an environment of competing goals between pilots and 
controllers, introduce instability into the air traffic control system in the form of increased 
negotiation and contention. The evidence does not seem to support this hypothesis. While 
there were instances of contention and extended negotiation, such instances were rare 
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when compared to the overall spirit of cooperation and teamwork between controller and 
pilot, even when cooperation meant acting contrary to their supposed competing goals. 

There is the possibility that the test subjects may have been predisposed to 
cooperative behavior. The test subjects for this experiment were unpaid volunteers and 
for the most part self-selected. As such, they may represent the more charitable, 
cooperative elements of their populations. In addition, knowing that their words and 
actions would be recorded and studied, subjects may have made an effort — conscious or 
subconscious — to be less egocentric and more synergistic in their problem-solving 
approaches. Furthermore, due to the close proximity of the cockpit and ATC simulators, 
the two subjects had the opportunity to become acquainted over the course of the day. As 
a result, the subjects tended to establish a friendly rapport that would not typify pilot- 
controller relations on the line. This rapport may have biased the subjects toward more 
cooperative, compliant behavior than is typical in actual operations. 

The availability of a NEXRAD weather overlay clearly benefited the controllers 
and the control system in general. Without the weather overlay, controllers had a difficult 
time anticipating the effects of weather on the traffic flow (i.e., building level 3 SA). As a 
result, controllers were faced with a high number of tactical deviations requiring time- 
critical conflict management. Attention to these immediate-term situations generally 
came at the expense of longer-term strategic planning. Furthermore, without good 
situation awareness regarding the location of weather-impacted areas, the controllers’ 
primary conflict resolution strategy was simply to meet the pilots’ re-route requests 
wherever possible. However, as suggested by the situation awareness analysis in Chapter 
2, the pilots requests typically reflected a desire to select the most efficient route that 
would avoid the weather; the impact of said route on the broader traffic flow was not an 
apparent goal of pilots. Thus, in attempting to honor pilots’ re-route requests, controllers 
were in effect subordinating their own goal of maintaining an orderly traffic flow to the 

pilots’ goal of selecting an efficient route. Ultimately, several separation violations 
occurred. 

When the weather overlay was provided, controllers were better able to anticipate 
aircraft needs and constraints, enabling them to shift their attentions from crisis 
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management and resolution to strategic planning and prevention. To varying degrees, the 
controllers adopted a more proactive role in routing aircraft around weather. Whereas in 
the non-datalinked configuration controllers typically waited for pilots to request 
deviations for weather and deferred to them for routings, in the datalinked configuration 
controllers often assigned vectors around weather in advance of any pilot requests. In 
such cases, pilots did not attempt to inject their goal of selecting the most efficient route 
into the re-routing decision. The controllers were free to select route amendments which 
optimized the overall traffic flow. In effect, this subordinated the pilots’ goal of selecting 
an efficient route to the controllers’ goal of maintaining an orderly traffic flow. No 
separation violations occurred in this datalinked configuration. These results illustrate 
how the allocation of information can influence the authority structure. 

One controller expressed that it was his goal to assign the vectors before the pilot 
asked for them, since the earlier the vectors were assigned, the more likely the pilot 
would be to accept them. Indeed, pilots accepted all of the controller-initiated weather 
vectors without contention, even when the vectors took them on a different routing than 
they had requested in the same scenario performed without the data link. Thus, the 
controller’s use of the weather information as a competitive advantage went unchecked 
by pilots, and the stability of the control system was not adversely affected. 

The markedly improved performance (in terms of separation assurance) and 
strong subjective preference of controllers for the weather display suggests that weather 
information of a quality equivalent to NEXRAD should be made available on the PVD. 
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Chapter 5 Conclusions 


It is generally thought that by sharing information between pilots and controllers, 
situation awareness will be improved on either side. With improved situation awareness, 
more collaboration between the two parties is anticipated. Such collaboration is expected 
to lead to improved performance on an individual and system-wide basis. 

The results of this study tend to corroborate the conventional wisdom. By sharing 
traffic and weather information between pilots and controllers, situation awareness with 
respect to traffic and weather was improved for both parties. Sharing of this information 
did lead to more collaborative interaction between the pilots and controllers, as evidenced 
by more frequent advisories and the unsolicited exchange of suggestions for alternative, 
more favorable routings. With improved situation awareness and increased air-ground 
cooperation, safety was improved, as evidenced by a reduction to zero in the number of 
separation violations. 

Outside the laboratory, the effect of shared information on pilot-controller 
interaction will depend on the degree to which pilots and controllers approach their work 
with the same spirit of cooperation as was evidenced in this study. When the pressures 
and realities of line operations begin to weigh on the pilot-controller relationship, it is 
possible that the spirit of cooperation may succumb to the more competitive, distributive 
interests identified in the situation awareness analysis associated with this study. In such 
cases, it is possible that by sharing information between the pilot and controller, re-route 
negotiations could become more protracted and more contentious. 

Independent of the effects of shared information on pilot-controller interaction, 
there appears to be a clear benefit to the provision of NEXRAD-type weather information 
to center controllers as an overlay on the PVD. Such displays appear to significantly 
improve controller situation awareness with respect to weather. More importantly, there 
appears to be a corollary benefit by which controllers are able to acquire better situation 
awareness with respect to traffic, particularly at the higher levels: comprehension and 
projection. In so doing, controllers appear to shift from reactive control strategies to more 
proactive ones, resulting in safer, more routine traffic operations. 
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In considering the addition of a NEXRAD weather overlay to the PVD, one 
significant benefit is the relative simplicity with which it could be implemented. The 
radar technology exists. The radar equipment is fielded and operational. The data 
dissemination networks are widespread and inexpensive (an air-ground data link is not 
required). Weather data products are commonly available; indeed, they are already found 
at most supervisors’ desks, just a few feet behind controllers’ heads. The remaining 
challenge is a relatively low-technology, low-risk one: integrating graphical weather 
products into the plan view display. 

The FAA’s proposed NAS architecture includes provisions for an improved 
graphical display of weather information for controllers. To accomplish this, current 
plans call for development and deployment of two new systems: a Weather and Radar 
Processor (WARP) and an en route ATC console called the Display System Replacement 
(DSR). The WARP will collect radar data and generate and disseminate a mosaic of 
NEXRAD images to the DSR for display with aircraft targets. 

Although the WARP is already operational at all U.S. ARTCCs, it will need to be 
upgraded to stage 1 in order to deliver NEXRAD imagery to the DSR (Kalani, 1999). 
The stage 1 upgrade is scheduled to be operational at the Seattle Center in 2000. 
Installation at the remaining Center facilities is planned for 2000-2001, by which time 
DSR also is expected to be operational at all U.S. ARTCCs (Johnson, 1999). The findings 
of this report argue that deployment of these systems be accelerated wherever possible. 

Providing controllers with a better picture of the weather situation certainly will 
not replace the need for air-ground data link. In fielding such systems in the future, it is 
hoped that designers will be mindful of the human components of the system, the 
different perspectives and interests that each brings to a shared problem, and how 
information can be employed to foster cooperative interaction and stability in the system. 


74 



References 


Amonlirdviman, Keith, Farley, Todd C., Hansman, R. John, Ladik, John F., & Sherer, 
Dana Z. (1998). A distributed simulation facility to support human factors 
research in advanced air transportation technology. SISO Simulation 
Interoperability Workshop, Orlando, Florida. 

Cashion, Patricia, Macintosh, Margaret- Anne, McGann, Alison, & Lozito, Sandra. 

(1997). A study of commercial flight crew self-separation. ATA A/TF.FF. Digital 
Avionics Systems Conference Proceedings, Irvine, California, October 26-30, 
Vol. 2, pp.6.3- 1 8 to 6.3-25. 

Endsley, Mica R. (1995). Toward a theory of situation awareness in dynamic systems. 
Human Factors, 37(1), pp. 32-64. 

Endsley, Mica R., Farley, Todd C., Jones, William M„ Midkiff, Alan H„ & Hansman, R. 
John (1998). Situation awareness information requirements for commercial airline 
pilots. ICAT-98-1, MIT International Center for Air Transportation, Cambridge, 
Massachusetts. 

Endsley, Mica R., & Rodgers, Mark D. (1994). Situation awareness information 

requirements for en route air traffic control. (DOT/FAA/AM-94/27), Washington, 
D.C.: Federal Aviation Administration Office of Aviation Medicine. 

Endsley, Mica R., & Jones, William M. (1997). Situation awareness information 

dominance & information warfare. AL/CF-TR- 1997-0 156, Wright-Patterson 
AFB, Ohio. 

FAA. (1998a). Federal aviation regulations. FAR 91.3, Federal Aviation Administration, 
Washington, DC. 

FAA. (1998b). FAA Order 71 10.65L, Air traffic control. Paragraph 2-10-lc.l.(a), Air 
traffic control handbook, Federal Aviation Administration, Washington, DC, 
February 26, 1998. 

FAA. (1998c). National airspace system architecture (draft). FAA Office of System 

Architecture and Investment Analysis (ASD-1), Federal Aviation Administration, 
Washington, DC. 

Falcone, Rich, Kollman, Kevin, Leber, Bill, Moffatt, John, Sandusky, Lorraine, Shanz, 
Art, Wambsganss, Mike, & Wetherly, Jim. (1998). Demonstrating an improved 
weather awareness system for CDM. Journal of Air Traffic Control, July- 
September, 1998, pp. 50-58. 

Foushee, Clayton H„ Lauber, John K., Baetge, Michael M., & Acomb, Dorothea B. 
(1986). Crew factors in flight operations: III. The operational significance of 
exposure to short-haul air transport operations. NASA Technical Memorandum 
88322. 

Hansman, R. J., Kuchar, J. K., Clarke, J. P., Vakil, S. V., Barhydt, R„ & Pritchett, A. R. 
(1997). Integrated human-centered systems approach to the development of 
advanced air traffic management systems. 1st USA/Europe ATM R&D Seminar, 
Saclay, France. 


75 



Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): 
Results of empirical and theoretical research. In Human Mental Workload, P. A. 
Hancock and N. Meshkati (Eds.), North Holland Press, Amsterdam. 

Johnson, Eric N. (1995). Multi-agent flight simulation with robust situation generation. 
Master’s degree thesis, Department of Aeronautics & Astronautics, Massachusetts 
Institute of Technology, Cambridge, Massachusetts. 

Johnson, Jay (1999). FAA AUA-460, WARP Product Team. Personal communication. 
January 13, 1999. 

Kalani, Nannette (1999). FAA AUA-460, WARP Product Team. Personal 
correspondence. January 13, 1999. 

Kanki, Barbara G„ & Prinzo, O. Veronika (1996). Methods and metrics of voice 

communications. (DOT/FAA/AM-96/10) Washington, D.C.: Federal Aviation 
Administration Office of Aviation Medicine. 

Lewicki, Roy L„ Saunders, David M., & Minton, John W. (1997). Essentials of 
Negotiation, McGraw-Hill. 

Midkiff, Alan H., & Hansman, R. John. (1992). Identification of important "party line" 
informational elements and the implications for situational awareness in the 
datalink environment. ASL-92-2, MIT Aeronautical Systems Lab, Cambridge, 
Massachusetts. 

Prinzo, O. Veronika, & Britton, Thomas W. (1995). Development of a coding form for 
approach control/pilot voice communications. (DOT/FAA/AM-95/15) 
Washington, D.C.: Federal Aviation Administration Office of Aviation Medicine. 

Pritchett, Amy R., Hansman, R. John, & Johnson, Eric N. (1996). Use of testable 
responses for performance-based measurement of situation awareness. 
International Conference on Experimental Analysis and Measurement of Situation 
Awareness, Daytona Beach, Florida. 


76 



Appendix A Commercial Airline Pilot Goal Hierarchy 


Basic Goal: Get aircraft from origin to destination safely, legally, 
with satisfactory levels of comfort and service to passengers, 
on schedule and in an efficient manner 


1.0 Select best path to destination 
(p. 86) 


— 1.1 Assess flight plan 

— 1.2 Determine changes 

to flight plan 

— 1.3 Replan flight path 


3.0 Manage resources effectively 
(p. 88) 


— 3.1 Tailor operations according 

to colleagues’ competency 
and reliability 

— 3.2 Keep everyone informed 

at appropriate level 

— 3.3 Tailor operations according 

to the status and reliability 
of the aircraft systems 
and airspace infrastructure 

— 3.4 Configure aircraft and 

determine system settings 
for phase of flight 

— 3.5 Configure aircraft and 

determine system settings 
for current weather and 
terrain 

— 3.6 Minimize impact of 

system abnormalities 


2.0 Execute desired flight path 
safely, efficiently, and 
with ride comfort 
(p. 87) 


— 2.1 Evaluate and execute plan 

— 2.2 Operate the aircraft 

to execute the plan 

— 2.3 Avoid conflicts 

— 2.4 Avoid hazardous weather 

and minimize impact of 
weather encounters 

— 2.5 Minimize impact of 

abnormal situations 

— 2.6 Minimize impact of 

abnormal ATC 
situations 


4.0 Satisfy the customer 
(p. 89) 


— 4.1 Provide for the safety 

and security of 
passengers 

— 4.2 Provide customer service 


Figure 56. Commercial Airline Pilot Goal Hierarchy (5 pages) 
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1.1.1 Insure safety and 
legality of flight 

— 1.1. 1.1 Avoid obstacles 

and terrain 

— 1.1 .1.2 Avoid restricted 

and special-use 
airspace 

— 1.1 .1.3 Avoid hazardous 

weather 


t 1.2.1 Assess costs and benefits 
of change 

1.2.2 Assess viability of 
potential flight path 


— 1.3.1 Determine need for replan 

— 1.3.2 Change destination 

— 1 .3.3 Change route 

— 1.3.4 Change speed profile 


— 1 .1 .2 Operate on schedule with 

minimum fuel consumption 
and acceptable ride quality 

— 1.1. 2.1 Establish priorities 

— 1. 1.2.2 Assess lateral flight path 

— 1.1. 2. 3 Assess vertical flight profile 

— 1.1. 2.4 Assess anticipated 

instrument arrival route 
and approach 

— 1.1. 2. 5 Assess taxi route and runway 

— 1.1. 2. 6 Assess departure plan 

— 1.1.3 Assess fuel sufficiency 

— 1.1.4 Determine impact of damaged, 

degraded or inoperable systems 
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2.0 Execute desired path safely, 
efficiently and with ride comfort 


2.1 Evaluate and execute plan 


— 2.1.1 Push back from gate 

— 2.1.2 Reassess departure plan 

— 2.1.3 Reassess taxi route 

and takeoff runway 

— 2.1.4 Takeoff 

— 2.1.5 Climbout 

— 2.1.6 Cruise 

— 2.1.7 Descent 

— 2.1.8 Hold 

— 2.1.9 Approach 

— 2.1.10 Landing and rollout 

— 2.1 .1 1 Taxi to gate 

— 2.1 .12 Go-around 

— 2.1.13 Shutdown 


2.3 Avoid conflicts 


— 2.3.1 Avoid obstacles and terrain 

— 2.3.2 Avoid restricted and 

special-use airspace 

— 2.3.3 Avoid other aircraft 

— 2.3.3. 1 Conform to ATC 

clearance and 
instructions 

— 2. 3. 3. 2 Identify potential 

intruders 

— 2. 3. 3. 3 Resolve traffic 

conflict 


2.5 Minimize impact of 
abnormal situations 


— 2.5.1 Bird strike 

— 2.5.2 Airplane off end/edge 

of runway/taxiway 

— 2.5.3 Fuel mismanagement 

— 2.5.4 Unusual attitude 
-2.5.5 Stall 

— 2.5.6 Overspeed 

— 2.5.7 CG mismanagement 

— 2.5.8 Off-airport landing 

— 2.5.9 Evacuation 

— 2.5.10 Passenger disruption 

— 2.5.1 1 In-flight medical emergency 

— 2.5.12 Hijacking 

— 2.5.13 Other abnormal events 


2.2 Operate the aircraft 
to execute the plan 


— 2.2.1 Determine appropriate 

level of automation 

— 2.2.2 Program navigation systems 

— 2.2.3 Maneuver aircraft within flight 

envelope for given configuration 

— 2.2.4 Maintain desired flight path 


2.4 Avoid hazardous weather 
and minimize impact of 
weather encounters 


— 2.4.1 Avoid hazardous weather 

— 2.4.2 Icing 

— 2.4.3 W ind shear/microburst 

— 2.4.4 Severe turbulence 

— 2.4.5 Thunderstorms and hail 

— 2.4.6 Heavy rain/snow 

— 2.4.7 Lightning strike or 

static discharge 

— 2.4.8 Low visibility 

— 2.4.9 Cold weather operations 


2.6 Minimize impact of 
abnormal ATC situations 


— 2.6.1 ATC outage 

— 2.6.2 Loss of communication 

with ATC 

— 2.6.3 Busted clearance 

— 2.6.4 Controller busy 

— 2.6.5 Other aircraft in distress 
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3.0 Manage resources effectively 


3.1 Tailor operations according to 
colleagues' competency 
and reliability 


t 3.1.1 Determine colleagues' 

competency and reliability 
3.1 .2 Determine impact of colleagues’ 
competency and reliability 


3.3 Tailor operations according to the 
status and reliability of the aircraft 
systems and airspace infrastructure 


t 3.3.1 Determine aircraft condition 
and equipment functionality 
3.3.2 Determine condition and 
functionality of air 
transportation system 
infrastructure 


3.5 Configure aircraft and determine 
system settings for current 
weather and terrain 


3.2 Keep everyone informed 
at appropriate level 


— 3.2.1 Advise cabin crew of schedule and 

safety issues (cabin readiness) 

— ■ 3.2.2 Communicate with maintenance 

— 3.2.3 Coordinate with dispatch 

— 3.2.4 Manage crew resources 
•—3.2.5 Inform ATC 


3.4 Configure aircraft and determine 
system settings for 
phase of flight 


— 3.4.1 Determine optimal 

engine start sequence 

— 3.4.2 Set configuration and system parameters 

for optimal takeoff performance 

— 3.4.3 Set configuration and system parameters 

for optimal climb performance 

— 3.4.4 Set configuration and system parameters 

for optimal cruise performance 

— 3.4.5 Set configuration and system parameters 

for optimal descent performance 

— 3.4.6 Set configuration and system parameters 

for optimal approach performance 

— 3.4.7 Set configuration and system parameters 

for optimal landing performance 

— 3.4.8 Set configuration and system parameters 

for optimal taxi performance 

— 3.4.9 Set configuration and system 

parameters for aircraft shutdown 

3.6 Minimize impact of 
system abnormalities 


— 3.6.1 Engine malfunction/fire 

— 3.6.2 Smoke in cabin 

— 3.6.3 System malfunction 

— 3.6.4 Cabin depressurization/ 

emergency descent 

— 3.6.5 Structural damage 

— 3.6.6 Flight control/surface failure 

— 3.6.7 FMS malfunction 

— 3.6.8 Autoflight malfunction 

— 3.6.9 Flight instrument/control 

contamination 
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Appendix B Commercial Airline Pilot Situation Awareness 

Information Requirements 


LEVEL 1 

Aircraft data 

• Call sign 

• Weight 

• Weight distribution 

• Center of gravity 

• Aircraft type 

• Engine type/capabilities 

• Equipment on board 

• CAT ll/lll qualified 

• First aid on board 

• Performance 
capabilities/restrictions 

• Maintenance 
carryover items 

Aircraft state 

• Heading 

• Magnetic 

• True 

• Altitude 

• Absolute altitude 

• Pressure altitude 

• True altitude 

• Density altitude 

• Temperature 

• Elevation 

• Altimeter setting 

• Airspeed 

• Indicated 

• Max and min 
airspeed for current 
configuration 

• Ground speed 
Airspeed rate of change 
Vertical speed 
Acceleration / 
deceleration 

Position 
Pitch attitude 
Roll attitude 
Turn rate 
Configuration 

• Gear position 

• Flap position 

• Slat position 

• Spoiler position 

• Stabilizer trim 


• Elevator trim 

• Thrust setting 

• Engines spooled evenly 
for takeoff 

• Fuel 

• Fuel quantity 

• Fuel temperature 

• Fuel type 

• Fuel distribution 

• Fuel burn rate 

• Arrival fuel 
requirement 

• Engine area 
clear/blocked 

• Braking force 

• Reverse thrust 

• Stall 

• Angle of attack 

• System settings 

• Anti-ice 

• Packs 

• Autopilot 
engagement 

• Wait time for de-ice 

• De-ice fluid 

• Type 

• Mix ratio 

Equipment malfunctions 

• Areas of aircraft 
damage 

• Operational status of 
aircraft systems 

• Reliability of 
systems 

• Severity of system 
failure/degrade 

• Validity of system 
failure/degrade 

• Pneumatic 

• Air condi- 
tioning 

. • Pressuri- 
zation 

• Thrust 
reversers 

• Hydraulic 

• Flight control 

• Flaps, slats 


• Control 
surfaces 

• Spoilers 

• Fuel 

• Electrical 

• Landing gear, 
brakes, anti- 
skid and nose- 
wheel, auto- 
braking 

• Navigation and 
instrumentation 

• Altimeter 
setting 

• Navigation 
system 
alignment 

• FMS pro- 
gramming 

• INS 

• GPS 

• ACARS 

• Powerplant 

• Engines 

• APU 

• Autoflight 

• Autopilot 

• FMS 

• Weather radar 

• Anti-ice 

• Fire protection 

• Audio panel 

• Communication 
system 

• Emergency 
systems and 
equipment 

• Oxygen 

• Smoke 
goggles 

• Doors 

• Unsafe 
exits 

• Lights 

• Warning 
systems 

• Cockpit voice 
recorder 
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• Operational status of 
ATC/NAS systems 

• System 
failures/degrades 

• Validity of system 
failure/degrade 

• Navigation aid 

• Communication 
system 

• Area of ATC outage 

• Command center 
outage 

• Confidence level in 
airspace systems 
functioning 

• Airport lighting 

Airports 

• Location 

• Altitude 

• Familiarity/recency 

• Closures 

• Altimeter setting 

• Active runway(s) 

• Approach in use 

• Runway information 

• Length & width 

• Weight restrictions 

• Surface conditions 

• Closures 

• Procedures in 
effect 

• Taxiway information 

• Width 

• Weight restrictions 

• Surface conditions 

• Slopes/grades 

• Closures/caution 
areas 

• Communication 
procedures 

• Alternate airport 

• Refueling 
capabilities 

• Tug capabilities 

• De-icing 
capabilities 

• Passenger 
accommodations 

• Customs 

• Stairs/jetway 

• Availability of 
medical care 

• Served by airline 


• Special information 

• Obstacles 

• Procedures 

• Noise 
abatement 

• Ground 
movement 

• Miss approach 

• Parallel 
approaches 

• Limitations 

• Landing curfew 

• Lighting/signage 

• Navigation ID and 
location 

Flight plan 

• Available routes 

• Available altitudes 

• Planned flight path 

• Distance 

• Altitude 

• Waypoints 

• Bearing 

• Discontinuities 

• Direction 

• Number of changes 
required 

• Difficulty of 
changes required 

• Dispatcher's 
concurrence with plan 

• Fuel reserve 
requirement 

• Arrival fuel requirement 

• Assigned runway 

• Takeoff plan/settings/ 
critical points 

• Planned airspeed 
profile 

• Planned climb profile 

• Planned cruise altitude 

• Planned cruise 
airspeed 

• Planned descent profile 

• Approach plan 

• Approach category 

• Reference speed 

• Initial approach 
altitude 

• Marker-crossing 
altitude 

• Final approach fix 
altitude 


• Decision height 

• Minimum descent 
altitude 

• Missed approach 
point 

• Scheduled time of 
arrival 

• Terminal/Gate 
assignment 

• Gate availability 

• Door for deplaning 

ATC 

• Appropriate ATC 
organization/frequency 

• Success rate of other 
aircraft requesting 
clearance 

• English proficiency 

• Local transition altitude 

• Status of: 

• Checklists 

• Procedures 

• Briefings 

Traffic 

• Traffic on taxiway 

• Traffic on runway 

• Traffic on final 

• Delays on ground 

• Number of aircraft 
holding ahead 

• Assigned sequence 

• Spacing on final 

• Expected Further 
Clearance time 

• Other aircraft 

• Aircraft type/ 
capabilities 

• Position 

• Altitude 

• Communications 
present 

• Altitude rate 

• Airspeed 

• Pilot competence/ 
reliability 

• TCAS instructions 

Terrain/Obstacles 

• Location 

• Height 

• Minimum altitudes 
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Weather 

• Area affected 

• Altitudes affected 

• Conditions 

• Temperature 

• Dewpoint 

• Precipitation 
(level and type) 

• Visibility 

• Ceiling 

• Wind 

• Direction 

• Magnitude 

• Rate of change 

• Altitudes 

• Gusts 

• Crosswind 
component 

• Darkness 

• Direction and speed of 
movement 

• Intensity and rate of 
change of intensity 

• Present ice buildup 

• Ice accumulation rate 

• Turbulence 

• Altitudes 

• Area 

• Intensity 

• Speed gain/loss reports 
from other aircraft 

• Wind shear location/ 
severity 

• Aircraft go-arounds 

• Airport conditions 

• Precipitation 
accumulation 

• Runway visibility 

• CAT ll/lll status 

• Minimums 

NAS 

• Special use airspace 

• Boundaries 

• Status 

• Activation level 

• Limits and 
restrictions 

• Navaid information 

• Frequency 

• Identifier 

• Availability 

• Course 


Clearance 

• Pushback clearance 

• Departure clearance 

• Taxi clearance 

• Position and hold 
clearance 

• Takeoff clearance 

• Clearance to transition 

• Descent clearance 

• Approach clearance 

• Landing clearance 

• ATC instructions / 
vectoring 

• Assigned heading 

• Assigned altitude or 
altitude restriction 

• Assigned time-to-fix 

• Assigned spacing 
or sequence 

• Assigned airspeed 
or airspeed 
restriction 

• Time by which to 
comply with 
clearance 

• Reporting points 

• Assigned runway 

• Assigned taxiway 

• Restrictions 

Passengers/cargo 

• Number 

• Cargo load 

• Weight 

• Hazardous material 

• Human organs 

• Serious illnesses/ 
injuries 

• Medical personnel on 
board 

• Sensitivity to descent 
rate 

• Cabin temperature 

• Cabin status 

• Seat belts on 

• Flight attendants 
seated 

• Carts stowed 

• Type/status of meal 
service 

• Provision status 

• Meals 

• Beverages 

• Pillows/blankets 


• Communications 
equipment 

• Movie 

• Points of interest 

• Hijacker(s) 

• Number 

• Profile 

• Demands 

Human Resources 

• Flight crew ability/ 
reliability 

• Cat ll/lll qualified 

• Experience in 
aircraft 

• Experience in crew 
position 

• Currency in aircraft 

• Familiarity with 
route and airport 

• Correctness of 
tasks executed 

• Cabin attendants 
ability/reliability 

• Experience 

• Number 

• Time on duty 

• Languages 

• Self (pilot) ability/ 
reliability 

• Fatigue 

• Stress 

• Attitude 

• Alertness 

• Time on duty 

• Workload level 

• ATC ability/reliability 

• ATC facility 

• Stress/workload 

• Flexibility 

• Ability/reliability of 

• Maintenance 

• Ground crew 

• Dispatch 

• Communication 
channels 

• Dispatch 

• Maintenance 
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LEVEL 2 

Aircraft parameters 

• Confidence level in 
aircraft systems 

• Deviation between 
aircraft state & aircraft 
limitations 

• Deviation between 
current attitude and 
desired attitude 

• Deviation between 
current gross weight 
and allowable gross 
weight 

• Deviation between 
aircraft state & planned 
settings 

• Severity of degrades 

• Margin to Vi 

• Airspeed relative to 
max turbulence 
penetration airspeed 

• Margin to stall 

• Validity of indications 

• Airspeed 

• Altitude 

• Fuel quantity 

• Stall 

• Electrical power 
demands 

Aircraft control 

• Required control inputs 

• Heading correction 

• Pitch correction 

• Thrust correction 

• Directional control 
responsiveness 

• Stability of approach 

• Available thrust 

• Ramp maneuvering 
requirements 

• Ability to abort / go 
around 

• Deviation between 
current maneuver and 
optimal maneuver 

Airport 

• Availability of suitable 
alternate 

• Ability to reach 
alternate 


Flight planning 

• Runway suitability 

• Taxiway suitability 

• Ability to reach 
destination 

• Deviation between hold 
time and safe hold time 

Flight plan conformance 

• Deviation between plan 
and 

• Optimal profile 

• Safety/legal 
requirements 

• Aircraft capabilities 

• ATC requirements 

• Fuel sufficiency 

• Schedule deviation 

• Track deviation 

• Heading deviation 

• Altitude deviation 

• Airspeed deviation 

• Allowable tolerance for 
deviations 

• Discontinuities in plan 

ATC conformance 

• Deviation from 
assigned 

• heading or vector 

• altitude 

• time-to-fix 

• spacing 

• airspeed 

• Conformance of 
clearance with 
expectations 

Traffic 

• Current separation from 
other aircraft 

• Trajectory of other 
aircraft relative to 
ownship 

• Closure rate 

• Other aircraft's intended 
actions/path 

• Maneuver 

• Aircraft 

• Timing 

• Type 

• Aircraft ahead 

• Spacing 

• Type 


• Wake turbulence areas 

Passengers/cargo 

• Comfort level 

• Safety 

• Unsafe exits 

• Urgency of medical 
needs 

• Hijacker(s) 

• Level of threat 

• Ability to meet 
demands 

Compliance with 
Regulations/Procedures 

• Compliance with noise 
abatement requirement 

• Compliance with 
Standard Instrument 
Departure (SID) 
requirements 

• Compliance with 
Standard Arrival Route 
(ST AR) requirements 

• Cleared to depart gate 

• Distance from special 
use airspace 

• Time until next 
communication needed 

• Controller's 
understanding of own 
intent/needs 

T errain/Obstacles 

• Relative distance, 
bearing and altitude 

• Min/max climb/descent 
rate to clear obstacle 

Priorities 

• Relative priority of 
safety, legality, comfort 
schedule, efficiency 

Customer satisfaction 

• Acceptable schedule 
deviation 

• Connection 
requirements 

Emergencies 

• Risk of hazard to 
passengers/crew 


86 



Weather 

• Confidence level in 
weather information 

• Timeliness of 
information 

• Hazard level 

• Takeoff minimums 

• Landing minimums 

• Potential for 

• Icing 

• Thunderstorms 

• Turbulence 

• Effectiveness of anti-ice 
measures 

• Path of minimum 
weather exposure 

• Deviation between 
current weather and 
projected weather 

• Relative distance and 
bearing to weather 
areas 

Impact 

• Of aircraft malfunction / 
damage / abnormal 
condition on: 

• Aircraft 
performance/safety 

• Aircraft stability/ 
control 

• Stopping ability 

• Flight plan 

• Operational 
parameters / 
system status 

• Procedures 

• Passenger/crew 
safety 

• Of weather on: 

• Aircraft 
performance 

• Fuel system 

• Aircraft control 

• Passenger comfort 

• Passenger/crew 
safety 

• Flight plan 

• Takeoff 

• Landing 

• Of traffic on: 

• Separation / safety 
of flight 

• Schedule 


• Of change in flight plan 
/ aircraft maneuver on: 


• 

Safety of flight 

• 

Legality 

• 

Schedule 

• 

Fuel usage 

• 

Ride quality 

• 

Passenger 


connections 

• Of 

deviations on: 

• 

Safety of flight 

• Of 

action on: 

• 

Hazard potential 

• 

Safety of flight 

• Of 

clearance on: 

• 

Safety of flight 

• 

Schedule 

• 

Efficiency 

• Of 

thrust level / 


configuration / system 
settings on: 

• Aircraft 
performance 

• Safety of flight 

• Passenger comfort/ 
safety 

• Fuel usage/ 
economy 

• ATC clearances/ 
restrictions 

• Of emergency on: 

• Safety of 
passengers/crew 

• Of conditions/flight 
status/information on: 

• Passenger comfort 

• Of automation on: 

• Safety of flight 

• Crew workload 

• Crew skills 

Workload 

• Time available to 
perform tasks 

• To execute change in 
flight path 

• Resources available 

• Utility of automation 

• Likelihood and cost of 
automation error 

• Time and effort to 
program and monitor 
automation 


• Time and effort to 
operate manually 

Cost / Benefit 

• Of change in: 

• Lateral flight path 

• Vertical flight profile 

• Takeoff runway 

• Departure route 

• Approach 

• Arrival route 

• Landing runway 

• Speed profile 

• Destination airport 

• Of holding vs. diverting 

• Of start/shut down of 
each engine 

• Of level of automation 

• Of evacuation 

Equipment malfunctions/ 
Aircraft condition 

• Deviation between 
system status and 
expected values 

• Deviation between plan 
and programmed 
automation 

• Impact of ATC 
degrade/outage on 
aircraft separation/ 
safety 

• Emergency status 

Human Resources 

• Confidence level in 
human resources 

• Flight crew 

• Self 

• Cabin attendants 

• ATC 

• Crew of other 
aircraft 

• Dispatch crew 

• Gate agent 

• Maintenance/ 
ground personnel 

• Areas of 
strength/weakness 

• Workload level 

• Ability to contain/calm 
unruly passenger(s) 
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LEVEL 3 

Aircraft 

• Projected trajectory 

• Own aircraft 

• Other aircraft 

• Projected relative 
trajectories 

• Projected separation 
between aircraft 

Flight plan 

• Projected taxi time 

• Projected schedule 
deviation 

• Estimated time of 
arrival 

• At destination 

• At fix 

• Projected fuel 
requirements 

• Predicted fuel usage 

• Predicted fuel burn rate 

• Projected time available 
on current fuel 

• Probability of ATC 
granting clearance for 
change in flight path 

• Probability of staying 
reliably on route 

• Predicted duration of 
hold 

• Predicted areas of 
congestion 

• Predicted periods of 
congestion 

• Predicted duration of 
delays 

• Predicted time: 

• On taxi 

• To departure 

• In each phase of 
flight 

• To destination 

• To alternate 

• In hold 

• To next clearance 

• Aircraft can safely 
remain in present/ 
anticipated 
conditions 

• Until maneuver 
required 


Weather trends/forecast 

• Projected hazard level 

• Projected area/severity 
of hazardous weather 
encounter 

• Predicted wind shear 

• Predicted turbulence 
along route 

• Predicted changes in 
visibility 

• Estimated time for 
weather to lift above 
minimums 

• Projected escape 
routes 

• Projected impact of 
changes/maneuvers/ 
weather on: 

• Safety of flight 

• Deviation from flight 
path 
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Appendix C Initial Briefing Materials 


Agenda 


1 . Welcome / Introduction 

□ Purpose of study 

□ Roles and responsibilities 

□ General structure of the session: test, survey, repeat 

□ Legal stuff 

□ A word about workload 

□ Questions? 

2. Simulator familiarization and practice 

□ What the simulator can and cannot do 

□ The en route airspace 

□ Special assumptions 

□ Questions? 

3. Six test runs 

□ Roughly 10 minutes each 

□ Workload survey 

4. Concluding tasks 

□ Summary workload survey 

□ Individual debrief 

□ Team debrief 
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AERONAUTICAL SYSTEMS LABORATORY 
DEPARTMENT OF 

AERONAUTICS AND ASTRONAUTICS 
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 


ROOM 35-217 

77 MASSACHUSETTS AVENUE 
CAMBRIDGE, MA 02139 
(617) 253-0993 FAX (617) 253-4196 


INFORMED CONSENT STATEMENT 


Research Assistant: 

Todd Farley 
MIT Room 35-217 
77 Massachusetts Avenue 
Cambridge, MA 02139 


Principal Investigator: 

Prof. R. John Hansman 
MIT Room 33-1 13 
77 Massachusetts Avenue 
Cambridge, MA 02139 


You may halt the experiment at any time and withdraw from the study for any reason 
without prejudice. You will remain anonymous in any report which describes this work. If you 
have any questions concerning the purpose, procedures, or risks associated with this experiment, 
please ask them. 


CONSENT 


In the unlikely event of physical injury resulting from participation in this research, I 
understand that medical treatment will be available from the MIT Medical Department, including 
first aid, emergency treatment and follow-up care as needed, and that my insurance carrier may 
be billed for the cost of such treatment. However, no compensation can be provided for medical 
care apart from the foregoing. I further understand that making such medical treatment available, 
or providing it, does not imply that such injury is the Investigator’s fault. I also understand that 
by my participation in this study, I am not waiving any of my legal rights 4 . 

I understand that I may also contact the Chairman of the Committee on the Use of Humans 
as Experimental Subjects, MIT 617-253-6787, if I feel I have been treated unfairly as a subject. 

I volunteer to participate in this experiment which is to involve using simulator computer 
displays for a total of four hours. I understand that I may discontinue my participation at any 
time. I have been informed as to the nature of this experiment and the risks involved, and agree 
to participate in the experiment. ° 


Date 


Signature 


Further information may be obtained by calling the Institute’s Insurance and Legal Affairs Office at 617-253-2822. 
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Participant No. 


PILOT BACKGROUND DATA 


Position: 
Current Equipment: 
Type Ratings: 
Total Time: 
Hours in current type: 
Glass Cockpit Hours: 
FMS Hours: 
Airline/Operator: 
Age: 
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Participant No. 


CONTROLLER BACKGROUND DATA 


Position: 
Current Facility: 
Previous Facilities: 
Years FPL: 
Years Radar: 
Years Non-Radar: 
Years Supervisory: 
Years Military: 
Years En Route: 
Age: 
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Appendix D Training Materials 


Pilot Briefing 

□ The 767-200 simulator 

□ Displays 

□ Primary flight display 

□ IAS appears to read low for the given mach 

□ Map display 

□ Range control (mouse) 

□ Navaid declutter button (mouse) 

□ Weather display 

□ No tilt: Assume identical returns at all altitudes 
(in other words, no opportunity to climb over or 
descend under the weather) 

□ Seven color radar return 

□ Traffic display 

□ All aircraft within +/- 2600 feet of your altitude 

□ Similar to current TCAS display: 

Relative altitude in 100’s of feet 

General traffic (white “vee”), Proximate traffic (white triangle), 

Traffic advisory (amber triangle), Resolution advisory (red triangle) 
“Traffic, Traffic” with amber traffic advisory 
Pitch command with red resolution advisory 

□ Aircraft call sign in gray 

□ Relative groundspeed in knots (minus indicates other aircraft is slower) 

□ Mode Control Panel 

□ LNAV and ALT HOLD are the initial autoflight modes for each scenario 

□ Full-time autothrottle 

□ Speed select 

□ Flight level change 

□ Heading select 

□ Altitude select 

□ Vertical speed select 

O Flight directors are non-functional 

□ Autothrottle and autopilot paddles are non-functional 

□ Flight Management System (limited functionality) 

□ LEGS page 

□ DIR INTC page 

□ Inserting waypoints 

□ Going “direct” 

□ Side-stick controller 

□ Autopilot engage/disengage switch (red button) 

□ Radios 

□ There is no radio management 

Assume you are automatically on the correct frequency all the time 
□ Push-to-talk switch (left shift key) 
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Pilot Briefing (cont'd.) 


□ The flight 

□ Airline revenue service flight 

□ You are roughly mid-way through a long-haul flight 
westbound: to LAX 

eastbound: to JFK 
northbound: to YYZ (Toronto) 
southbound: to Houston 

Note: In all cases, FMS will state your destination as JFK (ignore) 

□ Schedule performance and ride quality matter! 

□ Night flight (hence the dark out-the-window view) 

□ All scenarios occur in the cruise phase of flight, 
with aircraft trimmed and autoflight systems engaged 

□ All scenarios traverse the same en route airspace sector 
belonging to INDY CENTER 

□ Service ceiling for today’s gross weight: 40,000 feet 

□ Wind reports not available — assume zero wind 

□ Jeppesen map is provided — special provision: you and ATC 
are using victor airways despite being at cruise altitudes 

□ The scenarios 

□ Assume you have just been handed off by the previous sector; 

your first call should be to establish radio contact with new Indy Center controller 

□ Your goal is to balance safety of flight, schedule, ride comfort, and efficiency of 
operation 

□ For purposes of this study, you will be asked to verbalize your thoughts as best 
you can. You do not need to narrate what you are doing. Instead, try to “think 
aloud” as you evaluate the situation. This will help us understand what the 
important factors are in your decision-making process. 

□ Questions? 
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Controller Briefing 


□ The PVD simulator 

□ Sector display 

□ Brightness control 

□ Offset 

□ Overlays 

□ Weather display 

O NEXRAD: Assume identical returns at all altitudes 
(in other words, no opportunity to climb over or 
descend under the weather) 

□ Seven color radar return 

□ Traffic display 

□ Histories 

□ Data blocks 

□ PVD commands 

□ Flight plan readout (QF) 

□ Segmented circle (QP J) 

□ Route display (QU) 

O Data block offset 

□ Altitude clearance (QZ) 

□ Limitations 

□ No quick action keys 

□ No conflict alert 

□ Numerous unsupported functions (handoff, pointout, etc, etc) 

O Radios 

□ There is no radio management 

Assume you are automatically on the correct frequency all the time 

□ Push-to-talk switch (left shift key) 

□ The sector 

□ You are assigned to sector 1 1 , Indianapolis (Indy) Center 

□ Sector is 1 4000 feet and above 

□ Magnetic variation is zero degrees 

□ Zulu time is 0200z; Temp/baro settings are ISA standard day 

□ Wind reports not available — assume zero wind 

□ Sector map is provided - special provision: you and all aircraft traversing 
your sector are using victor airways despite being at cruise altitudes 
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Controller Briefing (cont'd.) 


□ The scenarios 

□ You will have a few minutes prior to each scenario to survey the sector, 
update your flight strips, and get the general flow of the traffic 

You will be expected to accept incoming aircraft (workload allowing), and verbally 
hand off departing aircraft to the next sector. 

□ If you need to coordinate with a neighboring sector, the student shadowing you 
will play the role of the other controller 

□ Flight strips are provided for your use. The flight plan information covers only the 
portion of each aircraft’s flight before, through, and after your sector. Assume all 
aircraft are long-haul, (i.e., no departures or arrivals) 

□ For purposes of this study, you will be asked to verbalize your thoughts as best 
you can. You do not need to narrate what you are doing. Instead, try to “think 
aloud” as you evaluate the situation. This will help us understand what the 
important factors are in your decision-making process. 

□ Questions? 
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Date: 

Session: 

Scenario: 

Coder: 


Appendix E Voice Coding Worksheet 



Figure 57. Voice coding worksheet template 
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Appendix F NASA Task Load Index Survey Materials 


Workload Rating Instructions 

We are interested not only in assessing your performance but also the experiences you 
had during the different task conditions. Right now we are going to describe the 
technique that will be used to examine your experiences. 

In the most general sense we are examining the "workload" you experience. Workload is 
a difficult concept to define precisely, but a simple one to understand generally. The 
factors that influence your experience of workload may come from the task itself, your 
feelings about your own performance, how much effort you put in, or the stress and 
frustration you felt. The workload contributed by different task elements may change as 
you get more familiar with a task, perform easier or harder versions of it, or move from 
one task to another. Physical components of workload are relatively easy to conceptualize 
and evaluate. However, the mental components of workload may be more difficult to 
measure. 

Since workload is something experienced individually by each person, there are no 
effective "rulers" that can be used to estimate the workload of different activities. One 
way to find out about workload is to ask people to describe the feelings they experienced. 
Because workload may be caused by many different factors, we would like you to 
evaluate several of them individually rather than lumping them into a single global 
evaluation of overall workload. This set of six rating scales was developed for you to use 
in evaluating your experiences during different tasks. Please read the descriptions of the 
scales carefully. If you have a question about any of the scales in the table, please ask me 
about it. It is extremely important that they be clear to you. You may keep the 
descriptions with you for reference during the experiment. 

After performing each task, you will be given a sheet of paper with six blank r atin g 
scales. You will evaluate the task by marking each scale at the point which matches your 
experience. Each line has two endpoint descriptors that describe the scale. Note that "own 
performance" goes from "good" on the left to "bad" on the right. This order has been 
confusing for some people. Please consider your responses carefully in dist in g uishin g 
among the task conditions. Consider each scale individually. Your ratings will play an 
important role in the evaluation being conducted, thus your active participation is 
essential to the success of this experiment, and is greatly appreciated. 


99 



RATING SCALE DEFINITIONS 

Title 

Description 

MENTAL 

DEMAND 

{Low/High) 

How much mental and perceptual activity was required 
(e.g., thinking, deciding, calculating, remembering, 
looking, searching, etc.)? Was the task easy or 
demanding, simple or complex, exacting or forgiving? 

PHYSICAL 

DEMAND 

{Low/High) 

How much physical activity was required (e.g., pushing, 
pulling, turning, controlling, activating, etc.)? Was the 
task easy or demanding, slow or brisk, slack or strenuous, 
restful or laborious? 

TEMPORAL 

DEMAND 

(Low/High) 

How much time pressure did you feel due to the rate or 
pace at which the tasks or task elements occurred? Was 
the pace slow and leisurely or rapid and frantic? 

PERFORMANCE 

{Good/Poor) 

How successful do you think you were in accomplishing 
the goals of the task? How satisfied were you with your 
performance in accomplishing these goals? 

EFFORT 

(Low/High) 

How hard did you have to work (mentally and physically) 
to accomplish your level of performance? 

FRUSTRATION 

LEVEL 

(Low/High) 

How insecure, discouraged, irritated, stressed and annoyed 
versus secure, gratified, content, relaxed and complacent 
did you feel during the task? 


100 




Participant No. 


Testcase No. 


MENTAL DEMAND 

i i i i i i i i I i i i i i i i i i i 


Low 


High 


PHYSICAL DEMAND 

I I I I 1 I I I I 1 I I I I I I I l 1 


Low 


High 


TEMPORAL DEMAND 

Li I i I i I i 1 i I i I i I i I i i 

Low High 


PERFORMANCE 

Li I i I i 


Good 


1-1 I I I I I I I I I I 


Poor 


EFFORT 

1 I I I I I I l 1 ■ 1 ■ I ■ I ■ I ■ I 

Low High 


FRUSTRATION 


—I 1 I I I I I 1 I I l I I 1 I I I I 


Low 


High 
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Workload Sources Instructions 


Throughout this experiment the rating scales are used to assess your experiences in the 
different task conditions. Scales of this sort are extremely useful, but their utility suffers 
from the tendency people have to interpret them in individual ways. For example, some 
people feel that mental or temporal demands are the essential aspects of workload 
regardless of the effort they expended or the performance they achieved. Others feel that 
if they performed well, the workload must have been low and vice versa. Yet others feel 
that effort or feelings of frustration are the most important factors in workload and so on. 
The results of previous studies have found every conceivable pattern of values. In 
addition, the factors that create levels of workload differ depending on the task. For 
example, some tasks might be difficult because they must be completed very quickly. 
Others may seem easy or hard because of the intensity of mental or physical effort 
required. Yet others feel difficult because they cannot be performed well, no matter how 
much effort is expended. 

The evaluation you are about to perform is a technique developed by NASA to assess the 
relative importance of six factors in determining how much workload you experienced. 
The procedure is simple: You will be presented with a series of pairs of rating scale titles 
(for example, Effort vs. Mental Demands) and asked to choose which of the items was 
more important to your experience of workload in the tasks that you just performed. 
Select the item that represents the more important contributor to workload for the specific 
tasks in this experiment. 

After you have finished the entire series we will be able to use the pattern of your choices 
to create a weighted combination of the ratings from that task into a summary workload 
score. Please consider your choices carefully and make them consistent with how you 
used the rating scales during the particular task you were asked to evaluate. Don't think 
that there is any correct pattern; we are only interested in your opinions. If you have any 
questions, please ask them now. Thank you for your participation. 
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Participant No. 


Which item in each pair was a more 
Significant source of workload? 


Physical demand 

□ 

Effort 

□ 

Temporal demand 

□ 

Performance 

□ 

Temporal demand 

□ 

Performance 

□ 

Frustration 

□ 

Temporal demand 

□ 

Effort 

□ 

Temporal demand 

□ 

Performance 

□ 

Temporal demand 

□ 

Frustration 

□ 

Performance 

□ 

Effort 

□ 


□ Mental demand 

□ Frustration 

□ Mental demand 

□ Frustration 

□ Physical demand 

□ Mental demand 

□ Physical demand 

□ Performance 

□ Physical demand 

□ Frustration 

□ Physical demand 

□ Effort 

□ Mental demand 

□ Effort 

□ Mental demand 


103 


s 



